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ABSIBACT 

This report of a project to design a set of training 
guidelines for planning, managing, and evaluating cooperative 
education programs describes briefly tie procedurjes used in 
developing the guidelines and model; discusses the various components 
of the planning, management, and evaluation process; and presents 
guidelines and criteria for designing an evaluation component and 
subsequent training Workshop. The report is intended to be useful to 
local education agency project staff involved in planning, managing, 
and evaluating cooperative education programs. There are nine 
chapters in the report: (1) A discussion of the need for training in 
program planning, management, and evaluation, (2) a description of 
the procedures used in the project to identify and examine a number 
of cooperative education programs nominated as being exemplary by 
State education agencies, (3) discussion of planning and managing 
education programs in general, (4) examination of special problems in 
planning and managing cooperative education programs^ (5) review of 
basic models (student outcome, program process, and policy evaluation 
models) for evaluating special programs and projects^ (6) 
presentation of the evaluation model developed by the project and 
suggested for cooperative education, (7) guidelines for planning and 
management components of cooperative programs, (8J guidelines for the 
evaluation model components, and (9) format, guidelines, and sample 
-materials for a training workshop on the model. (Volume II, bound 
separately, contains appendixes: Materials on project management^ 
objectives, evaluation, and followup collected from the exemplary 
programs that were examined as background for developing the model.) 
(HD) 
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PROJECT ABSTRACT 



The objective of this project was to design a set of training 
guidelines for planning, managing, and evaluating cooperative education 
programs. In addition, a set of guidelines and criteria for designing a 
evaluation component and subsequent training workshop are described. 

The design specifications of the procedures build upon general education 
planning, management and evaluation models developed by the Center for the 
Study of Evaluation, (UCLA); Education TURNKEY Systems, (SPEMS) ; and 
portions of models or components developed and applied in a limited number 
of cooperative education programs which were nominated as being exemplary 
by SEAs. 

Even though existing report is initial phase of a three phase study 
(development and field test application) , report will be useful for LEA project 
staff involved in planning, managing, and evaluating cooperative education 
programs; will assist in developing staff training in directly related 
areas; and will assist new LEA staff as orientation information. 

Topics covered in materials include: Need for Training; Description of 
Exemplary Programs; General Planning and Managing techniques for Special Projects; 
Problems in Planning and Managing Cooperative Education Programs; An 
Evaluation Model for Cooperative Education; Evaluation Model Components. 
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BACKGROUND OF THE PROJECT 



Funded by the U.S. Office of Education under Part C, Vocational 
Education Research, this project represents a limited attempt to design 
a set of guidelines for training program directors in planning, managing, 
and evaluating cooperative education projects. The original proposal in- 
cluded a design phase similar to that described in this volume; the 
development of the model and training materials; a field test involving 
practitioners; and final packaging of the model and training components. 
However, due to a 75% reduction in budget, a much more limited effort was 
negotiated with the U. S. Office of Education in June 1975. It was antici- 
pated at that time (and plans have been proposed) that the project would 
continue along the lines described above in developing a validated package 
of procedures and training material. A proposal has been submitted to 
develop and apply the generic model, with additional developmental efforts 
to post-secondary cooperative education projects with a special focus on 
measuring productivity. In addition, the existing component (evaluation 
questions) and additional procedures are being proposed for field testing 
in a public school system's cooperative education programs during the school 
year 1976-77. In short, the information provided in this document should 
be considered a preliminary design intended to be used as a basis for future 
developmental efforts. 

In light of the above limitations, however, it was felt that the pre- 
liminary design and materials provided in this volume would be useful for a 
number of purposes: to various individuals responsible for aspects of coopera- 
tive education. 



First, State education officials may note our findings and recom- 
rnendations in determining their priorities for providing assistance to 
cooperative education programs in their states. Our review of existing 
practices in cooperative education projects at the local level indicated 
a significant gap or deficiency in the area of evaluation- A recent 
study of urban cooperative education programs conducted by Olympic 
Research was highly critical of cooperative education program operations 
In the area of evaluation^ It should be noted that this deficiency has 
also been recognized by the Government Accounting Office; and various 
versions of legislation address themselves to the questions of evaluation 
and accountability in vocational education generally at the State and 
local levels. This would seem good evidence to support State efforts 
to develop and implement improved evaluation systems for cooperative 
education at the local level. Our specifications provide the first step 
for such a developmental effort. 

Second, in discussions with project staff involved in cooperative 
education programs at the local level and officials within SEAs, it be- 
came increasingly clear that an administrative rather than managerial 
philosophy exist. Where emphasis is placed upon administering a program 
to meet minimal requirements, many opportunities are lost in terms of in- 
novation, creativity, and productivity due to the lack of a managerial 
philosophy and approdch to the problems associated with cooperative 
education. Hence, it was felt that our general introduction to the con- 
cepts of Project Management and Evaluation could be used to foster 



Vincent Breglio, Marta Steven & Jeanette Tobias, An Assessment of Sch ool 
Supervised Work Education Programs , Olympic Research--Decima Research, 
Santa Ana, CA. o 



a different perspective at the local level which would be the foundation 
for further improveinents in the quality and effectiveness of projects. 

Third, local and regional agencies interested in developing exemplary 
projects could adopt the general model, or specific design features, to 
meet local needs and proceed with the development of field testing phases. 
The specifications were designed with this type of flexibility in mind, 
to allow for adaptation to varied circumstances (e.g. secondary/post- 
secondary, urban/rural). 

Before describing the preliminary design of the model, it would appear 
appropriate to review very briefly some of the existing models which were 
reviewed in the process of synthesizing and developing the overall design. 
Since many of these models are fully developed, interested readers may 
wish to obtain information on these models if they appear to be appropriate 
for application in their respective projects. 

First, during the initial phases of the study (during the summer of 
1975), the project team reviewed the planning and evaluation models developed 
by several regional educational labs involved in the experienced-based 
career education model. These particular procedures and components have 
been under development for over 3 years and reflect an enormous amount of 
time and resources directed by the labs utilizing funds from both the U,S, 
Office of Education and later from the National Institute of Education, 
For the most part, these models and components were directly relevant to 
the area of cooperative education. However, due to the experimental and 
demonstrative aspect of the operational components which were to have been 
evaluated, the cost of the planning and evaluation components for these 
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experienced-based models appear to be extremely high and nonfeasible for 
implementation by a large number of districts involved in cooperative 
education programs (especially the medium size to smaller districts). 
However, it should be noted that subsequent to our analysis, the guide- 
lines for fiscal year 1976 for Part D, the Vocational Education Act as 
amended, indicated a priority concern in having local districts (where 
appropriate) propose to utilize components from the experienced-based 
models in their Part D proposals ar;j subsequent projects. In tenns of 
timing, if for no other reason, it was felt that any description of these 
models in a report of this nature would be too late and unnecessary since 
descriptive im'onnation was provided to applicants by the U.S. Office of 
Education program monitor of Part D projects.^ 

Second, we reviewed a number of specific components of existing 
models which appeared to be relevant, a few of which are mentioned below. 
The Delphi technique applied at the Dallas Independent School District in 
the planning phase of the Skyline Institute appears to be a very effective 
and reasonable approach for projecting long-term occupational demands 
within a metropolitan area. The approach for determining the cost-- 
effectiveness used by Battel le Memorial Institute (Columbus, Ohio) 
for cooperative education programs is extremely useful from a policy 
point of view; however, it has limited applicability at the local level. 
A model presently under development by the U.S. Office of Education 
(OPB&E) conducted by Development Associates, proposes an evaluation model 
'for career education projects.^ This model provides basic information re- 
garding the concept of evaluation and alternative models presently used. 
In addition, it provides extremely useful information regarding specific 



^For additional information contact Program Monitor, Part D Projects, 
USOE/BOAE, 7 and D Street, S.W., Washington, D.C. 
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procedures which can be applied under various conditions within various 
evaluation designs. However, this model is not directed toward co- 
operative education programs. Over the last 5 years. Research for Better 
Schools, Inc., a regional education lab based in Philadelphia, has been 
developing a model for planning, managing, and evaluating special pro- 
jects in education generally. This model (which is based upon earlier 
work conducted by Dr. Desmond Cook) is extremely sophisticated and re- 
quires an enormous amount of staff training for implonentation. While 
it still remains one of the most comprehensive models and training pro- 
grams for special projects in education, generally, little specific guidance 
is provided for cooperative education projects. 

Third, perhaps one of the most useful models far adapting to this 
particular project, is a model developed at the Center for Study Evaluation 
(UCLA) on program planning and evaluation over the last 5 years. This 
particular model, published by McGrav;-Hill , includes 'not only systematic 
procedures but also an intensive training component. This model appeared 
to be extremely useful for cooperative education and its problems since 
the major focus is upon evaluation with program plaraiing and management con- 
sidered subcomponents of an overall evaluation model. Recent variations of 
this CSE model have been published by the authors of the CSE model and in- 
dividuals involved in this specific project. 

And last, given the problems identified during our survey, components 
of the TURNKEY SPEMS model appear to be directly relevant. This particular 
model is similar to the RBS model and the CSE model in certain respects; 
however, it is much simplier than the RBS model, dir«:ted toward a learner 
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without much knowledge and experience in planning, managing ^ and evaluating 
special projects. Under development since 1S71, thfisB particular model has 
been developed jointly with practitioners from over S5 school districts 
across the country reflecting a practical and pragmaitrc rather than a 
theoretical approach. It was felt, however, that tte evaluation component 
of SPEMS was too technical and not directly appl icacCTe^ to cooperative 
education programs. 

In summary, the reader is reminded that this psTticular model should 
be considered a preliminary design with inherent 1 imriHations imposed by 
budget constraints and the need for an extended time for the completion 
of development, field testing, and packaging efforts^ It was felt, how- 
ever, that give', the national attention to cooperatt.vs education and the 
glaring deficiencies in certain areas (such as evaluation) that the in- 
formation in such a preliminary design would be usefuTi to a large number 
of practitioners at the State and local level. 
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CHAPTER I 
THE NEED FOR TRAINING 



At present, cooperative education is extremely popular; 
President Ford voir-^^ ri support for the expansion of work experience 



in August 1974. i che General Accounting Office, in an otherwise 

heavily critical examination of vocational education, endorsed 
cooperative education programs, saying their use ought to be 



doubled in size between 1970 and 1974 — 605,140 students, up from 
2 

298,915 — that is quite an endorsement. 



The National Manpower Institute, in its report The Boundless 

Resource, released in November 1975, recommended the development 

of programs giving all students at least 500 hours of work or service 

experience and the institution of a comprehensive program of 

3 

community internships and work apprenticeships. 



Comptroller General of the United States, U. S. General Accounting 
Office. Report to the Congress: What is the Role of Federal 
Assistance for Vocational Education? , Washington, D. C; U. S . 
General Accounting Office, December 31, 1974, p. 91, 

Data from the U. S.. Office of Education, Bureau of Occupational 
and Adult Education. In 1974, 115,34 5 students were reported in 
programs funded under Part G and 489/795 in programs funded 
under Part B of the Vocational Education Amendments of 1968. In 
1970, 23,001 were funded under Part G and 266,914 under Part B. 

^National Manpower Institute. The Boundless Resource: A Prospectus 
for an Education-Work Force Policy . November 1975. 



programs in hir .pe 



1 work and education at Ohio State University 



expanded further. 



Since cooperative education programs more than 
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Beatrice G. Reubens, in her paper prepaared for the HEW 
Secretary's Committee on Work in America, stated "a desirable goal 
for cooperative education might be 50% of alii in high school 
vocational and general education . - 

However, amid all this praise, a warnit>5 also emerges. There 
are so^^^ cant weaknesses in the desigir and management of 

cooperauxve programs, and unless these flaws are rectified, the 
cooperative education movement could find itself consigned in five 
or ten years to the category of education nKoStels that promised 
more than they delivered. 

A major weakness in cooperative prograitrs today — as in many 
education programs — is in management and evaluation. Specifically, 
managers of cooperative education programs sE^ldom systematically 
evaluate their activities to discover which girogram elements work 
well and which do not. Right now, enthusiasm is high. Informal as- 
sessments and intuition seem adequate to sujgort the conviction that 
these programs are effective. But that can ^nardly last. The com- 
petition for funds at all levels of governmecEt promises to be in- 
creasingly intense in the coming decade and tthe anticipated expansion 
of cooperative education programs will undotfe^edly be followed by a 
demand for formal evaluation. By and larger, managers of these pro- 
grams are not now prepared to meet this challenge. 



Beatrice G. Reubens, "Vocational Education for All in High School?", 
prepared for the Secretary's Committee on WaTk in America, 
Department of Health, Education , and Welfare,, September 1972. 
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Developing management capability, particularly with respect 
to evaluation, is critically important for the cooperative education 
movement today for at least two reasons. First, such capability 
will make managers of cooperative education programs better able to 
detect early those aspects of their programs which require bolstering 
and those which could be expanded. Second, such management 
capability will crive these programs a firmer base from which to 
bid for scarce resources in the future, when the predisposition of 
funding sources toward cooperative education may be less favorable 
than at present. 

The evidence that evaluation is a major area of management 
deficiency in cooperative education and related types of work ex- 
perience education seems clear. 

In order to measure the success of cooperative education pro- 
grams, one needs follow-up data on the success of program graduates 
in obtaining and progressing in employment. Yet many programs do 
not collect follow-up data or do so in an informal manner which 
does not allow one to draw inferences about the program. In an ex- 
tensive study of fifty exemplary cooperative education programs con- 
ducted in 1973, Frankel found that only 61% had any follow-up pro- 
cedures at all, and the vast majority of those were smaller programs. 
Only 8% of programs with 40-99 students had follow-up procedures and 
none of the programs with more than 100 students collected follow- 
up data . ^ 



^Steven M. Frankel, et al.. Case Studies of Fifty Representative Work 
Education Programs . System Development Corp., Santa Monica, Ca., 1973 



Frankel also found that program admin isttr a tors seemed to have 
little understanding of what attributes of taneir programs contributed 
to success. For example, when surveys of th© opinions of students, 
employers, and administrators concerning the success of cooperative 
education programs were compared, no correlation was found between 
the ratings administrators gave their program:s and the satisfaction 
indicated by students and employers. In facttr/ on some indices 
high marks by administrators on program qualilty had negative 
correlations with successful outcomes.'^ 

Investigations of vocational education iprrograms in general 
have revealed a widespread lack of modern mamagement and evaluation 
procedures. The 1974 General Accounting Offikre report stated, 
"In the states we visited the existing vocatikonal programs at all 
levels lacked adequate student follow--Tip. We were told that with- 
out this type of information, 1) it is extreinsly difficult to 
determine the extent to which specific traiatiBg is impacting on 
individual and labor market needs, and 2) eas^tial information 

on which to base instructional changes is not available to vocational 

7 

educators and planners." . 

A statewide study of the evaluation of Itocal vocational 
education programs done by the Bureau of Vocattional Education in 

^Ibid. 

"^Comptroller General of the United States, op^ cit . 



Kentucky found "not only was there a lack of formal organization, 

but there was no formal evaluation procedure . . • Those responsible 

for operating vocational education programs;, especially at the 

local level, did not possess the expertise Ib evaluation and 

organization necessary for effective vocatioaial education 

evaluation The study found that local program managers 

needed to know how to organize a successful evalua'i .ioa . and 

also needed training in what evaluative procedures and techniques 
8 

to use. 

Informal discussions with educators at the state and local 
level who are concerned with cooperative education programs 
confirmed the findings of these studies, nan^ly, that there is a 
general lack of sophistication in management techniques on the 
part of program administrators and that the greatest need for 
training is in the area of evaluation. 

Therefore, in the development of a model and guidelines for 
training materials for cooperative education program directors, 
the following principles were followed: 

1) the major emphasis should be on developing basic 
capability in program evaluation; 

2) the evaluation model used should be practical, easy 
to understand, and suited to the special needs of 
cooperative education ; 



%loyd C. McKinney, Alfred J. Manneback, and C. 0. Neel, Final 
Beport. Central Kentucky Vocational Education Evaluation Project , 
bureau of Vocational Education,. State Department of Education, 
ZFrankf ort , Kentucky. 
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3) the planning and management instruction should build 
on current capabilities and be developed to increase 
effectiveness . 
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CHAPTER II 
EXEMPLI T>Y PROGRAMS 



Within the limited scope and budget of this- project, efforts 
were made to examine the best of current practice in planning, 
management, and evaluation of cooperative education programs. 
State Directors of Vocational Education in forty states were each 
asked to nominate two cooperative education programs within their 
states that had noteworthy planning, management, or evaluation 
components. The program director at each noijiinated site was then 
sent a letter explaining the project and requesting relevant in- 
formation and documents . 

The following definitions of cooperative education, work-study, 
and work experience were used (these definitions appear in the 
letter to the site directors) : 

COOPERATIVE EDUCATION: a program of on-the-job 
work experience related to the student's course 
of study and chosen occupation. Such programs 
have educational objectives, including specific 
skill training objectives, and the work ex- 
perience is closely tied to classroom in- 
struction. 



^For copies of the letters sent to and responses received from 
State Directors and site directors, see Appendix A. 
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WORK-STUDY: a program of employment to pro- 
vide financial assistance to students who are 
in need of earnings from employment to com- 
mence or continue their vocational program. 
The employment is not necessarily related to 
the student's course of study. 



WORK-EXPERIENCE: a program of - on-the-job work 
experience designed to acquaint the student 
with, the job setting. Such programs have 
educational objectives — usually in the area 
of expanding the student ' s career horizons or 
introducing the student to job activities 
and requirements — but not skill building 
objectives, A wide variety of programs, in- 
cluding most career education work units 
and post- secondary clinical experience pro- 
grams, fall into this category. 



The materials sent in response to this inquiry had much. to 
recommend them. However, it also became clear that even these 
exemplary programs rarely had systemmatic management and evaluation 
procedures, A review of the materials thus tended further to 
support our conclusion, expressed above, that cooperative education 
programs wonld benefit from increased use of formal management 
techniques with special emphasis on evaluation. 

A word of cantxon should be expressed here in interpreting 
the following findings. Because funds were not available for site 
visits or extensive follow-up, the project staff was solely 
dependent on materials submitted by the site directors. The 
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intention was to examine, as simply as possible, the best of current 
practice, rather than to attempt an indepth analysis of 
representative programs. 

The programs from which responses were received reflected 
many commonalities amid great variety. All the programs examined 
were true cooperative education programs with on-the-job work 
experience and a required related class. In most cases the 
related class was taught by the same teacher/coordinator who 
arranged and supervised the work experience. A number of sites 
also had additional, informal work experience programs for 
students who did not qualify for the cooperative education program. 

Most programs required a minimum of fifteen (15) hours a week 
on the job, with classes in the morning and work in the afternoon. 
The range of allowable work hours per week was from ten (10) 
(Visalia, California) to forty (40) under special circumstances 
(Fort Wayne f Indiana). As a variation from this typical format of 
classes in the morning and work in the afternoon, New York City 
offers an alternate week program in which students attend school 
full time one week and work full time the next. 

The most common program areas were distributive education, 
industrial education, and office occupations. Most of the programs 
were designed for relatively highly employable students. 

Common student selection criteria included: 

1) the student must be 16 years of age and a senior (a 
few programs admitted juniors); 
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2) the student must be at grade level in basic subjects 
and be accumulating sufficient credit to graduate; 

3) the students must be rated by their teachers, councelors, 
and principals as reliable, punctual, honest, and 
successful in dealing^ with peers and adults; 

4) the student must evidence a sincere interest in the 
program and be able to benefit from it. 

One program manual specifically stated that the cooperative 
education program was '*not for disruptive students". 

Several sites had special programs for students with special 
needs. Mesa, Arizona, for example, has a Career Lab for potential 
dropouts. Industrial Occupations for students with special needs, 
and Occupational Education for students with learning problems. 
Only one site, Visalia, California, specifically integrated 
special needs students into the regular cooperative education 
program, Visalia had as an objective that 35% of the enrollment 
be minority, handicapped, or disadvantaged students. 

Required teacher/coordinator's supervision of the work site 
ranged from a mandatory visit of one-half hour per student once 
every two weeks (Hanover Park, New Jersey) to monthly visits 
(Providence, Rhode Island) to a minimum of twice per quarter 
(Visalia, California) , 

The materials submitted by the exemplary sites supplied 
considerably less detail regarding management procedures than was 
provided regarding program content. 
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PROJECT MANAGEMENT 



Frequently the program management scheme was presented in 
terms of the coordinator's job description. Almost all sites sent 
detailed descriptions of the coordinator's responsibilities. In 
the following pages are typical examples from Milwaukee and New 
York City^ (see Appendix B for details) . 

Examples of Coordinator ' s Responsibilities : 

Milwaukee : 

• Identifies and Selects Program Student/Learners 

• Describes the program to students 

• Works with teachers, guidance counselors, admi.nistrators 

• Provides occupational information 

• Gathers information on students 

• Schedules programs for student/learners 

0 Counsels student/learners and their parents 

• Assists student/learners with career planning 

• Identifies and Selects On-the-Job-Training (OJT) Stations 

• Enlists participation of cooperating employers 

• Identifies suitable training stations for each 
student/learner 

• Orients employers, training supervisors and co-workers 

• Prepares students for job inter*views 

• Assists student/learners to obtain placement in on-^ 
the- job- training stations 

• Prepares and processes training agreements 

• Assists Student/Learners to Adjust to the "World of Work" 

• Assists student/learners with problems which arise 
in the OJT situation 

• Confers with training supervisors regarding "adjust- 
ment" problems 

• Evaluates the student/learners progress in the OJT 
situation 



Department of Elementary and Secondary Education, Division of 
Curriculum Instruction, Milwaukee Public Schools. Industrial 
Cooperative Education Operational Guidelines , 1972, pp. 5-7. 

Bureau of Cooperative Education, Office of High Schools, Board 
of Education of the City of New York. Cooperative Education 
Coordinators Handbook , 1974, pp. 9-14. 
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New York: 

• Administrative Responsibilities to the Cooperative 
Education Bureau: 

• Attend monthly meeting of Coordinators at the 
Board of Education 

• Attend meetings with the Borough Coordinator 

• Submit records and reports in accordance with dates 
specified in monthly Cooperative Calendar ... 

• Make supervisory visits to industry as directed by 
the Central Office, and submit report 

• Administrative Responsibilities Within the School — 
Principal and Assistant Principals: 

• Consult the administration concerning school 
schedules for Cooperative students ... 

• Suggest to Assistant Principals (Supervision) 
teachers who will relate well to Co-ops 

• Involve the Assistant Principals iSupervision) in 
the program and request that they recommend potential 
Co-ops for the following term. This is especially 
helpful in the business area. 

• Discuss with Assistant Principals (Supervision) 
curricular modifications based on current techniques 
and trends in industry. Relay information gained 
from your visits to industry and from students' job 
experiences . 

As part of an experimental management system, the Glendale 
Union High School District has refined a description of the 
coordinator's responsibilities and tied each task to expected 
completion dates. Although the management documents are not in 
final form (the project to develop the management system is still 
in progress), they have been included in Appendix B as an example 
of this approach. 

Three sites included management flowcharts in their materials. 
They have been reproduced in Appendix B. The presence of flowcharts 
or other devices for planning and monitoring program process does 
not necessarily mean, of course, that those programs are better 
managed than are programs without such tools. However, as cooperative 
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education programs expand, schools may find that an informal 
management approach is not sufficient to produce effective control. 
The time task chart in one of its infinite varieties is a simple 
tool which can help the program director keep track of activities 
and assure that tasks are being completed on time. The versions 
shown demonstrate three possible approaches which schools have 
found useful. 

OBJECTIVES 

All of the sites had objectives. All had either student 
outcome objectives' or program process objectives. Bellevue, 
Nebraska had the following student outcome objectives among others 
described in Appendix C.^ 

• The student will possess skills and positive attitudes 
towards performing his assigned tasks. 

• The student will represent the business favorably to 
customers and outside business associat 

• The student will be able to demonstrate x'xitiative and 
creativity in selected problem solving situations. 

• The student will be able to talk clearly and pleasantly, 
conveying spirit and enthusiasm. 

• The student will become aware of other's needs and 
motivations in order to work more cooperatively with 
fellow employees, supervisors, and management. 

• The student will develop the attitude that personnel 
policies are established for the benefit of the store 
and the employees . 



Taken from information supplied by Vocational Director, Secondary,* 
Regular Cooperative Programs, Bellevue, Nebraska. 
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Visalia, California had the following program process objectives 



for one of its programs (also see Appendix C) • 

• During the 1975-76 school year^ cooperative agriculture 
education will be provided for a minimum of twenty 
students of whom at least eighteen will acquire saleable 
skills necessary for employment. 

• By the end of 1975-76 school year, each student enrolled 
in the program will have had an opportunity to complete 
175 hours of related classroom instruction. 

• By the end of the 1975-76 school year, each student 
enrolled in the program will have been provided an 
opportunity to complete a minimum of 35 0 hours of paid 
cooperative work experience. 

• During the 1975-76 school year, the teacher/coordinator 
will continue to use an individualized, self-pacing, 
continuous progress curriclum. 

• By December 1975, at least 80% of the students enrolled 
in this program will be members of the Future Farmers 
of America (FFA) . 

• Each student who completes the program will have developed 
job entry level skills necessary to retain employment 

and to be certified employable by the teacher/coordinator 
and the cooperative work experience sponsor. 

• By June 1976, the teacher/coordinator will have attended 
at least five hours of work experience education in- 
service training. 



ERLC 



Vocational Education Project Application for Funds Under the 
Vocational Education Amendments of 1968, State of California- 
Vocational Education, Visalia Unified School District, p. 15. 
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The following are examples, taken from Providence, EHode 

6 7 
Island and Livingston, New Jersey , respectively which demonstrate 

the most commonly found type of objectives — those which describe 

the kinds of opportunities to learn the students will have: 

• Develop an awareness in the student of the real relation- 
ship that exists in the world of work and learning in 
school thereby maintaining the student to seek further 
growth and development which will enhance his future 
life style; 

• Develop in the student individual qualifications for 
subsequent full-time employment or advanced study ; 

• Encourage the student to continue his schooling by 
permitting him to satisfy his financial needs through 
part-time employment ; 

• Enhance the student's occupational preparation by 
involving him in the real world of work. 

• To provide students with specific vocational training 
in order to provide for articulation between classroom 
and job; 

• To have the student examine the necessity of good human 
relations and put them into practice; 

• To allow the student to put his classroom learnings to 
practical use; 

• To provide an opportunity for the development of 
certain desirable work habits such as industr iousness , 
responsibility, self-reliance, and punctuality; 

• To provide opportunities and experiences for the develop- 
ment of social skills and the ability to get along with 
others » 



Cooperative Vocational Education Guidelines, Cooperative Vocational 
Education Program of the Providence School Department, Providence, 
Rhode Island, June 1975, p. ^2. 

''cooperative Office Education Manual, Livingston High School, 
— ^Liv-lngs-ton^— New— Jer^seyr— P-. — 2-. — 
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It is easier to plan, manage, and evaluate programs which 
have clearly stated and distinct student outcome objectives and 
program process objectives. None of the materials received showed 
both. 



EVALUATION 

There are three kinds of evaluation problems addressed in 
the materials: 

1) evaluation of the student's performance during and at 
the conclusion of the program; 

2) student success in obtaining and holding jobs in the 
year following graduation; and 

3) general program evaluation. 

Providence, Rhode Island reported one of the more extensive 
lists of ways in which student achievement is measured: 

• Evaluation cvC student knowledge and applications 

• teacher developed tests 

• standardized tests 

• teacher observation 

• practical tests on equipment 

• On the job performance as evaluated by 

• weekly attendance sheets 

• monthly employer evaluation reports 

• monthly coordinator visits to job sites. 

Evaluation of the student's performance on the job is done 
primarily by the employers in most sites. Some sites also use 
teacher/coordinator ratings and student self -ratings . Examples of 
the forms used by several sites for this purpose are included in 
Appendix D. 
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FOLLOW-^ 

Seven sites reported a follow-up system in place (or planned) 
to ascertain the success of students in obtaining employment after 
graduation. Arizona sites reported that the state performed all 
follow-up studies. Indiana sites sent information about a new 
"System for Implementing Review and Follow--Up (SIRF) " developed by 
the State. Wyoming, Michigan transmitted both the State follow--up 
form and a special form used in a State Advisory Council Study 
(see Appendix E) . 

Providence, Rhode Island reported that all 1974 cooperative 

education graduates were surveyed a year after graduation and the 

following pattern was found: 

176 completed the program 

71 employed full time in field trained or related field 

17 other employment 

10 unemployed 

11 higher education 

12 not available for placement 
55 status unknown 

New York City provides information comparing the attendance 
and dropout rates of cooperative education students with citywide 
rates and also comparing the attendance, lateness, and academic 
achievement records of cooperative education students with records 
they established before entering the program. In all cases the 
compaxisoir^- fevored the cooperative education program. 

Gene3^ -rites reported informal program evaluations, usuaULy 

conststinrr of descriptions of the programs in action and suggestions 

£or^ahang==^^. The„eval.uations„jv.er.e_de3^eiap^ed_b^ 

on an advisory committee. 
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In no sites were there evaluations reported which compared 
the skills or job placement rates of cooperative education students 
with those of a comparable group of non-cooperative education 
students. In no sites were there reported systemmatic evaluation 
studies which linked specific aspects of cooperative education 
programs with differing levels of student achievement. 

To be sure, our examination uncovered a number of useful 
devices at the exemplary sites which could serve as models in a 
training program in the planning, management, and evaluation of 
cooperative education programs. For example, some of the employer 
rating forms and follow-up forms which we have reproduced appear 
quite useful. 

However/ this examination lends weight to the conviction that 
cooperative education programs could benefit from rigorous manage- 
ment and evaluation training and the systemmatic application of 
these skills. 
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CHAPTER HI 

PLANNING AND MANAGING EDUCATION PROGRAMS: GENERAL 

Management improvement in public education has increasingly 
become a priority at local, state, and federal levels. Only five 
years ago a survey conducted by the American Association of School 
Administrators found that only 1% of superintendents felt skills 
in management areas such as "systems analysis" would help them 
administer or run a school. Of the 50 "topics" scheduled for the 
AASA Annual Convention in 1976, 36 are directly related to 
managemenli. Moreover, practically all "job descriptions" for 
super intendency openings identify "management skills and experience" 
as a prerequisite. Indeed, the number of superintendencies and 
other high level staff positions filled by "professional Managers" 
has increased dramatically. 

This demand for betl:er management comes from several forces. 
Public -elementary and secondary education is . in a recession with 
student enrollments declinirrg for the first time in tMe history 
of this country; this requirres new administrative ski!LM: and 
perspectives. Inflation rates, intensified by the enercy crisis, 
iave had a psrrricularly higti impact on public schools. Increased 
xjrganization and unionizaticcn of "labor" Jias forced a zimnge in 
management approach. And state, federal^^ and local maziaates for 
accountability have forced the need to "managre for re mits in the 
70's". 
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While the need for improved management generally is increasing, 
the demand for effective management for special projects and 
programs is even greater. 

FUNCTIONS OF MANAGEMENT 

The functions of management in education differ significantly 
from those generally accepted and used in industry. In the private 
sector, particularly industry, the function of management is to 
plan, organize, direct, and control an operation, usually wL±h the 
objective of maximizing profit in one of two ways: a) given a pre- 
determined expense budget, maximize sales volume, or b) gi^en a 
predetermined or contracted volume of sales, minimize costs. 

In education generally, the functions of management are 
similar to those above; however, the goals are significantly 
different. Conceptxially , an analogy could be to "maximize the 
production of leading , and skill acquisition given a projected 
budget"; however/ ir reality goals are ^Idom output oriented but 
rs±:her prx)cess orzi^tied. For example, at the simplistic level 
grven many of line above factors contributing to the economic 
depressirm in ^^Sunatlon today, the goal of many school managers 
JLS to keep schncl doors open and programs operating. In a limited 
number of cases , Jiowever , an output orientation for management does 
-exist, using measnres such as student scores on national 
standardized achievement tests. For the most part, however^ the 
goals of education management fall between these two extren^s. Yet, 

34 

-19- 



due to increased pressures for accountability, the movement 
generally is toward an output orientation. 



MANAGEMENT REQUIREMENTS FOR SPECIAL PROGRAMS 

The major management tasks prior to and during the operational 
piiase of programs include a) program planning, b) program prepara- 
tion and startup, c) program monitorring, and d) problem identifica- 
tion and problem solving. 

A. Program Planning 

The management tasks in planning a program are identified 
below although they do not necessarily have to follow sequentially 
in all situatiQXis. 

Planning usually begins with rhe identification of 
problems jx disrrirepancies between actual progress in an area and 
the stated goals of the district. Once the discrepancies are noted , 
then the specific goals of the program :are esta±)lishiad . These 
goals are aL^o usually established throirgh a consensus process^ 
attempting ~ identify the causes rather than symptomis of the 
proisiems . 

If lacal resources are not available^ then outside 
resources to fund a program designed to solve the identified 
prabtems must be found. Here, sources might include foundations, 
state education agencies , and appropriate divisz:^onH 'qz federal 
aqmicz^^s. Mxc:. in maaiy instances the nature of prograim guideLin-es 
vmOiZ. consxxain to soitK extent the natuo^^ of the probZten as defined 
and the types of appn^aches which are acceptable fnr inclusion in 
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the proposal generated by the district. These proposals usually 
include the identification of problems, the justification and 
their verification, the discussion of alternatives and various 
constraints, seljsction of the alternative which makes the most 
sense and appears to be most cast-effective^ concurrence of staff, 
and then approval by the district. 

Once tie overall goals and objectives of the program are 
identified, therL the major task is to determine what activities 
(e.g., specific actions which c:onsume resources) are necessary. 
This is usually done by breakirirg the overall program, into work 
packages, which axe activi±ies trimt a^B s±milHr in substance 
(e.g., test adminxstratioii: and test smrdrtg) or are conducted by 
similar staff (e.g ^ Instrr ircrxoii in niatti aird instruction in 
reading in an eLem^ritaixy sr±ool , both x:nTid:ar:ted by the home room 
teacher) . 

Once the :^ecir±c: ta^Jcs and a:cti*::3Lties are identified, 
then they are arrsn^^ed in such a way that :the flow of work can be 
displayed, usually visually and in a sequerrtial and logical order. 
An example of s-iich a "work flow" diagram Ib presented in Figure 1. 
In developing z work flow, one is const,:.ntly attempting to minimize 
peaks and trGm;^^ in tb:e use of limited resources over an extended 
period of time to prevent "nown time" cz: staff. 

Aftsr "activities ha^ been ifemrified, and usually 
concurrent wi±h the develqpnneiif: of a "\iKork: flow" , the program 
H Ir "e c t o r ^"and' s±a':f f ~^de vel^op- -^^^ s t iina t^^^^ xtf-xe s our c e s^ii s e d - 1 o 
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FIGURE 1 
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Selected Events 

A. Project begins 

Program operations begin 
(instruction) 

C. Program operations end 

D. End o£ project 



Date 

6/73 
9/73 

6/74 
9/74 



Tasks 

1. Determine objectives 

2. Select teachers 

3. Select tests 

4. Select materials 

5. Purchase materials 

6. Prepare materials 

7. Develop" instructional program 

8. Field test the program 

9. Develop training program 

10. Conduct teacher training 

11. Administer pre- test 

12. Score pre-test 

13. Conduct program operations 

14. Administer post- test 

15. Score post-tests 

16. Determine and report results 
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Represents an event, the start or completion o£ an activity or 
the completion o£ one activity and tlie start o£ another. 

Represents a time or resource consuming activity conducted 
between t\^^o events. 



Represents activities which do not consimie time or resources; 
e.g., transfer o£ in£onnation or results o£ previous event into 
subsequent tasks • g y 
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actually implement the program which in turn: is usually reflected 
in a budget • An illustration of processes for depicting the total 
amount of resources used in conducting specific tasks is illustrated 
in Figure 2 . 

B. Program Preparation and Startup 

An evaluation design must be developed and/or refined to 
assure that appropriate processes and proceSures are utilized to 
determine whether the project objectives arsf being met. Such a 
design may have been already developed durirng the planning phase. 
If so, it should be re-examined before being implemented, especially 
if program modifications occurred during n^ptiations with the 
funding agency. 

An information system must be dev^oped which will 
accommodate the evaluation design and managment monitoring 
activities during the operational phase of me program. It must be 
determined whether the use of the existing ^Mstrictwide information 
reporting procedures can be used; to the extent they cannot, 
additional reporting procedures which are specific to the program 
need to be developed. The specific data system requirements will 
have to take into account who will decide wlnat, when, and on what 
basis. In most instances, instruments will have to be developed 
for collecting, processing, and analyzing inoformation about the 
program during its operational phase and refflucing it into a 
format suitable for decision making purposes. 
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A number of startup activities involved in most special 
programs include: 



• 



the selection, hiring, and training of staff; 



• the development and/or refinaaent of an overall 
organizational chart which delineates lines of 
communications, responsibilities, and authorities 
of the program director as well as individual 
program staff assigned to con^flmct specific activities; 



# 



purchasing of materials, supplies, equipment, etc.; 

orientation to program staff as well as officials 
interested in or affected by the special program. 



It should be emphasised that the effectiveness by whib-H 
program preparation and startup activities are conducted by the 
program director is usually the single most important factor in 
the determining of the overall success of tbe program. 
C. Program Monitoring 

Once the program is implemented, a major responsibility 
of the program director is to monitor the program through personal 
site visits as well as through a predetermined reporting system. 
As described later, to the extent the program director is perceived 
by staff as a "problem solver", monitoring -will be viewed in a 
positive rather than a negative way. The monitoring and reporting 
systems used in special programs vary considerably; however, to 
have an effective system one should include instruments which 

a) clearly identify the persons responsible for completion of 
specific tasks and the criteria and date for completion of the task; 

b) an early warning mechanism to flag slowdowns; c) an instrument 
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which depicts (visually in most instances) the progress of all 

tasks being undertaken in light of the objectives to be achieved 

and related utilization of resources. 

D. Problem Identification and Problem Solving 

The ultimate purpose of monitoring a program is to assure 

its completion by the early detection of problems and the 

application of effective procedures for solving them. Problems may 

be created by external factors such as funding agency redirections 

and/or internal conditions such as personnel turnover. The 

program director must constantly consider the trade-offs (i.e., 

what has to be abandoned in one area to get something in another) 

invclving a) time constraints, b) resource availability, and 

c) the levels of performance. 

The process of problem identification and solution is 

at the heart of managing special programs. The specific steps 

are noted below: 

Step 1 Identify Problems: 
Verify existence, 

Determine what is not the problem. 
Determine degree of severity (tolerance) , 
Rank problems aiccording to impact. 

Step 2 Determine Causes: 

Assess program data. 
Consult with staff, 
Simulate the cause. 
Verify. 
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step 3 Analyze Alternative Solutions (Opportunities) 
Verify potential 
assess data 
assess context 

identify strengths and weaknesses 
Rank alternatives with staff consultation 
Test "best" alternative 

time and costs 

meet objectives 
Select alternatives 

Step 4 Execute Solution: 
Document, 
Communicate, 
Follow through. 
Modify plan. 
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CHAPTER IV 

SPECIAL PROBLEMS IN PLANNING AN© MANAGING 
COOPERATIVE EDUCATION PROGRAMS 

The following discussion is intended to provide perspective 
on special problems in planning and managing cooperative education 
programs and give some direction to those designing training 
materials dealing with these issues. 
ISSUES IN PLANNING 

A. Clarifying Goals 

As examples of the kinds of problems planners face 
in defining the objectives of cooperative ed^acation programs, we 
will examine briefly three commonly found objectives of these 
programs. 

The most common major objective of cooperative 
education programs is to prepare students for employment. The 
objective might be stated as follows: Upon leaving school, all 
students who desire to do so shall obtain appropriate paid or 
unpaid employment or continue with further education. In line with 
this, the placement rate of program graduates in jobs becomes an im- 
portant component of program evaluation. Yet there are two dangers her 
which should be understood from the start. The. first danger is 
confusing the ability of the school to traim workers with the 
ability of the economy to provide jobs. Training materials should 
urge educators to keep in mind that the general health of the 
economy and the unemployment rate have more effect on job placement 
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success than do training programs. Program planners should clearly 
understand that at best, the school can marginally increase job 
placement rates of youth by providing students increased access to 
existing jobs. Educators can help overcome the prejudice of 
employers toward young people. Also, they can give a small group 
of special program students a competitive edge over other 
youngsters their own age in securing employment. But in stating 
their objectives, program planners should be careful not to accept 
responsibility for economic problems beyond their control. 

This consideration is particularly important if one plans 
to expand a cooperative education program which has had a very 
high placement rate in the past. Training materials should urge 
program planners to ask themselves to what extent is it possible 
to maintain that placement rate if one doubles the number of 
students? Are there really enough jobs to absorb them or will they 
just be competing with each other for too few slots? 

The second danger with the job placement objective 
is that insufficient care will be given to placing students in 
good jobs. Although educators must be sensitive to the needs of 
employers, the training materials should make clear that it is not 
the purpose of the school to provide business and industry with 
willing workers at all levels. The purpose of the school is to 
increase the opportunity for students to obtain better jobs than 
they would have obtained had there been no job training program. 
"Better" is defined as being superior on one or more of the 
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following measures: 1) more satisfying personally, 2) brighter 
future for advancement, 3) more satisfying financially. 

Using the criterion of good jobs rather than just 
jobs leads to an additional complication — the different potential 
of different students. In planning a program one must define from 
the start what students one is talking about. Most cooperative 
education programs set minimum standards for enrollees, such as 
good attendance records and at least moderate academic accomplish- 
ment. Other programs are specifically designed for disadvantaged 
or handicapped students. The kind of job which w^uld constitute a 
successful placement for, as an exauiDle, a iBoderat^ly retarded 
stioiienl: will be quite different from a succ^sstfixl placement of a 
studemr qualified for intellectually challengCTj work. 

This must be taken into account whsi designing the 
program and obtaining training stations and also when planning the 
evaluation. Training materials must make the point clearly in the 
planning section, and must make it again forcefully in the 
evaluation section. In evaluating program success, students in 
the program should be compared to that of other students with 
approximately the same ability and background. Comparing the 
success of program graduates who are carefully selected from the 
top 30% of their class with the success of average students will 
obviously give results biased in favor of the program, while 
^ comparing the success of disadvantaged program participants with 
that of the average student will yield results biased against the 
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program. Obvious as these problems seem, such unfair comparisons 
are very common in evaluations of cooperative education programs. 

Consider another common type of objective for 
cooperative education programs: "Students will be able to make 
rational career decisions based on knowledge of self and knowledge 
of the world of work.". This objective seems reasonable enough at 
first glance — - after all, one does not want to encourage students 
to make irrational decisions — but if one Icroks at it up close it 
can be very trinky. What is a rational :^c±sion? How can one 
iBEasure it? I£ one measures rational chrxice by compar±ng students' 
yoh choices wife: their scores on traditional career inlsrest 
inventories, fox example, one has the problem of taking: into account 
inherent cultural bias and sex bias in the tests which will tend 
to direct minority group members and women into lower paying jobs. 
Career guidance tests, and even most guidance counselor interviews, 
can only measure interests that have already been developed. They 
cannot predict future interests or measure the ability of students 
to develop skills they have never tried. Rather than focusing 
entirely on matching students to jobs on the basis of current level 
of skill and int:erest, training materials should urge educators 
to be equally crmcerned with encouraging students to try new 
occupational areas. The materials should also deal in depth with 
the special problems that women and members of ethnic minorities 
face in choosing and obtaining jobs. 
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A third common type of objective is of the form: 
"Students will possess skills necessary for obtaining entry level 
jobs.". If planners have done their work properly in examining the 
implications of the job placement objective discussed above, 
planning for this objective becomes simple. Having decided what 
■jobs one wants txD prepare students for, the planner must determine 
what skills emp-Loyers require in employees applying for those jobs. 
iSome employers want workers who are reliable and punctual, can read, 
write and do sinrole arithmetic, can speak English clearly and have 
a serious attitude toward work. Given an employee like that, they 
prefer to do their own specific skill training. For other jobs, 
employers requiire specific skills. In some areas where employers 
have access to an experienced labor force tSey will not hire 
students just out of school who have not already acquired a skill. 
The major advantage of cooperative education programs, of course, 
is that frequently the student is trained daring the education 
program at the job and in the firm that could hire him or her at 
the completion of the program. So long as that job fits the 
"better" job definition discussed above, the problems of skill 
analysis are obviated. However, the materials should remind educators 
that many students will not be hired permanently by the employer 
with whom they did their training. Program planners therefore 
need to analyze the extent to which the training provided matches 
the requirements of the employer who will ultimately do the hiring. 
Furthermore, in planning the in^-school component of the program 

48 

-32- 



educators need to know which aspects of preparation for work 
should be emphasized. 

B. As^ssing Participants' Desires 

A cooperative education program works best when it 
suits the needs of both students and employers. For small programs 
this match is usually not too difficult to obtain. A certain 
percentage of local employers will be willing to participate. A 
certain number of students will be interested in training for the 
jobs being offered. Problems can occur, however, when a successful 
program expamis. Most commonly the problem is finding enough 
training stations for all the students who wish to participate. 
'Those in charge of the program may be tempted to accept slots that 
are really dead-end, low-paying jobs, inappropriate for an 
educational program. Likewise, the program should not accept 
students who do not need or do not want training but rather, for 
example, want an opportunity to earn money while attending school. 
Such students should be offered a time-release Oir work-study 
program and not enrolled in a cooperative education program which 
has specific vocational training objectives. 

Thus , in assessing participants ' desires , program 
planners need to be careful to compare those desires with the 
objectives of cooperative education and be sure that cooperative 
education is the program that is called for to meet those needs. 

Training materials should also emphasize that in 
planning for cooperative education programs, more than for most 



educatrLonal programs, it ±s important to crmsider the needs and 
desires of a variety of community members. Planners must assess the 
desires of parents, of employers, of labor union representatives, 
of organized interest gro:nps and of the ccCTaunity at large. One 
commonly used way of obtaining continual input from interested 
non-educators is to appoint an advisory CDimcil. 

In many school districts there anie craft advisory 
councils for each occupational area. Thermsmbers of these 
councils can be consulted by program planners. A single, district- 
wide advisory council can also be consulted. Training materials 
should include specific procedures for maintaining liaison between 
the council smd prograit^ managers throughout the year. 

Having ascertained what type of programs students 
desire and what types of programs the commxmity considers desirable, 
the planner is left with the job of surveying employers to 
estimate potential training stations. If there is a great demand for 
a training program in metallurgical technology but there are no appro- 
priate facilities nearby willing to participate in the cooperative 
education program, there is no point in proceeding with the plan. On th( 
other hand, planners should not get discouraged too easily. Seeking 
out training stations is a long, hard job. 
C . Labor Market Projections 

Simultaneous with the assessment of student and 
community desires, and of the availability of appropriate training 
stations, the planner must obtain the best possible projections 
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of future job openings. There is no point in preparing students 

for jobs which are likely to disappear • The planner should look 

at both likely local jobs and the job picture nationwide. 

Cooperative eduCa-t^ion, like all vocational education, should be 

closely tied to the labor market when possible. Although student 

desires are the primary concern of the school , the school also has 

the responsibility to provide students with the best possible 

information about the employment reality he or she will face after 

graduation, and not to continue to provide training in areas where 

there are no jobs. 

The problem of obtaining good projections is a 

difficult one. What data do exist are frequently in a format 

which is difficult to translate into educational terms. In the 

1974 report of the National and State Advisory Councils on Vocational 

Education, 70% of the state councils polled reported that the 

absence of current, accurate data in useful format severely 

1 

inhibited the planning process m their states. 

ISSUES IN PROGRAM MANAGEMENT 

The crucial difference between managing a cooperative 
education program and managing most other education programs is the 
relative lack of control. The educator must depend on the employer 



National and State Advisory Councils on Vocational Education. 
T he Impact of the Vocational Education Amendments of 1968 . 
National Advisory Council on Vocational Education, Washington, 
D. C, April 1974, pp. 4-5. 
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to provide the learning experience and must spend much of his or 
her time in liaison and coordination work. 

The key management concern in dealing with all the 
different parties involved in a cooperative education program is 
communication. Information about the program director's expectations 
must be clearly communicated to staff, students, participating 
employers, parents, advisory groups, and all other involved parties. 
Good, clear statements of objectives, definitions of the 
responsibilities of staff, students, employers, and the advsiory 
council can simplify this task considerably and avoid later 
misunderstandings . 

Even the best laid plans — and the best written job 
descriptions and employer agreements — are sometimes inadequate. 
Unexpected problems arise. At those moments a good communications 
system can minimize the difficulty and allow the program to proceed. 
Information about the problem must reach the program director, or 
other appropriate staff member , quickly and accurately. With good 
information in hand, the director can then check with the 
appropriate people and take necessary action. 

A good informal communications network should complement, 
but not substitute for, a formal data collection system. The 
training materials should make clear that only a well planned, formal 
data collection system can yield the type of information necessary 
for program documentation and evaluation. 

On the other hand, the formal data collection system, and 
the on-going evaluation, can yield information which can help the 
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program director make decisions concerning program changes in the 
course of the year. Obviously, if information comes back that 
there are serious problems at one particular work site or with a 
particular staff member, the program director will want to take 
appropriate action. Increased information means increased ability 
to keep the program on track. 

In addition to the particularly heavy burden of coordinat 
ing extensive off-campus activity, the director must also bear all 
the standard management responsibilities of program oversight. 
Overseeing the timely completion of tasks and the efficient use of 
resources is made easier if a clear management plan with an easily 
read visual display, such as a flowchart, has been designed at the 
planning stage. The director is then forewarned when dates by 
which specific tasks must be completed are approaching and helps 
the director see what the alternatives are when modifications are 
called for. 

The cooperative education program director also has a 
responsibility to maintain liaison with advisory councils and with 
community groups. The cooperative education program can only 
continue to succeed with strong community support. The training 
materials should emphasize that public relations work is an integral 
part of the program director's job and should not be slighted. 

Volume II contains guidelines for the development of 
training materials for directors of cooperative education programs 
which reflect the considerations discussed in this chapter and in 
the previous chapter on general problems in planning and managing 
special programs and projects. 53 



CHAPTER V 

EVALUATING SPECIAL PROGRAMS AND PROJECTS 



Educat^ion evaluation is rapidly emerging as a discipline of 
its own due to a number of factors. A variety of problems as well 
as approaches proliferate in the field as growing pangs naturally 
occur. Hence, below we describe the overall purposes of education 
evaluation, the reasons for its emergence at this time, typical 
categories of problems which have surfaced and general evaluation 
functions. Then, we discuss some of the specific problems as well 
as alternative solutions regarding evaluation of education 
programs. 

PURPOSES OF EVALUATION IN EDUCATION 

Education evaluation has two major purposes: a) to determine 
the extent to which the objectives of an education program are met; 
and b) to attempt to explain why such results occurred with the 
purpose of providing feedback for program improvement purposes. 
The emphasis upon the first purpose is often influenced by 
accountability pressures.. The latter purpose should always be a- 
priority in any evaluation. It is the former purpose of evaluation 
(sometimes called "product" evaluation) which tends to create 
anxieties, which are often unduly justified. The latter purpose 
of evaluation (also referred to as "process" evaluation or 
"formative" evaluation) is an integral part of project planning and 
management with program improvement intentions. 
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FACTORS CONTRIBUTING TO GROWTH OF EVALUATION 

Education evaluation as a discipline has grown dramatically 
over the last two decades due to a number of reasons many of which 
are related. 

Firsts the growth of evaluation generally can be attributed 
to the obligation to assess how well Federal and state funded 
education programs are succeeding. Generally speaking, its growth 
is liighly correlated with the initial passage of the Elementary 
and Secondary Education Act and other directly related programs 
such as the Vocational Education Act, as amended, even though 
the time lag has varied among programs. As the growth of education 
programs funded by Federal agencies, particularly non-formula 
type programs such as ESEA Title I, reached their plateau in the 
early 70' s, additional pressures to expand evaluation came from 
several forces: those who wanted to demonstrate the programs were 
successful and continue funding, from those who wanted to cut 
back programs, and/or from those who wanted to have explanations 
for both the success and failure of programs for policy formulation 
purposes. As an example of the heightened interest in evaluation 
on the part of Federal policy makers in Congress, the Education 
Amendements of 1974 mandated approximately $50 million to evaluate 
compensatory education alone. A similar heightened interest with 
congressional mandates has occurred in the area of vocational 
education, as described in detail later in this section. 



Second, concurrent with the natural obligation to evaluate 
increased numbers of education programs carae the pressures for 
accountability at all levels which further bolstered the demands 
for evaluations and in many cases to be conducted by independent 
third parties. In other instances where evaluations were conducted 
by school staff, requirements were made to have a third party 
conduct an independent and objective education program audit 
designed to assess the project and the evaluation design, certifying 
the degree to which the findings were indeed valid and reliable. 
In other instances, program and financial audits conducted by 
governmental agencies such as the Government Accounting Office 
further attenuated the recognition by districts that good evalua-- 
tions tended to reduce the number of "surprises" and reduce the 
burdens associated with outside audits. 

And last, a factor contributing not only to the growth but 
professionalism within evaluation were the initial findings during 
the late 60 's which provided justification to question many of the 
"sacred cows" in education. For example, the Coleman stiidy and 
subsequent re-analyses of data indicated that schools and the 
resources allocated to schools did not explain as many differences 
in student growth as did family related factors. Additional studies 
and evaluations of "successful programs" were for the most part 
inconsistent and the degree to which findings were generalizeable 
to schools across the country was certainly limited. Without 
doubt the focus upon questions related to "w^hy" fostered the growth 
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of the social science research movement in this country. As an 
example of this growth, the U. S. Office of Education has increased 
its funding for evaluations of education programs from approximately 
$1.2 million in 1968 and to $24 million in 1975. 

GENERIC APPROACHES TO EVALUAT ION 

While the details of an evaluation design should always be 
unique, reflecting the specific characteristics of the particular 
project to be evaluated, there do exist several generic models as 
summarized below. An extremely useful reference which summarizes 
a much larger number of models for various types of application 
(e.g., curriculum, training, program) can be found in a recent book 
entitled. Contemporary Appro aches to Program Evaluation authored 
by Dr. Sara Steele and published by Capitol Publications, 
Washington, D. C. 

A. The Stud ent Outcome Model 

This model is designed to determine how well a specific 
number of students (referred to as the "target population") 
achieved predetermined objectives in a subject matter area as a 
result of a specific progre.m or technique sometimes called an 
"intervention" or "treatment". Usually in this model there is some 
unit of comparison either in terms of a control group consisting 
of a number of similar students receiving a dissimilar treatment 
(usually the one regularly operated within the school or district). 
In other instances a unit of comparison may be the past performance 
of students; hence the comparison of the treatment is to expected 
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rates of learning or skill acquisition of the student or group. 



Perhaps the most common unit of comparison is determined by the 
use of nationally standardized norm-referenced tests which indicate 
how the target group compares to the national average. When the 
treatment or technique being tested such as in a pilot program 
occurs over a specific time period, in many instances the same 
instrument is administered on a pre-treatment basis and post- 
treatment basis to determine the net gain. Sometimes an attempt 
is made to explain why the students did or did not achieve their ' 
objectives or not as well as the control or comparison standard. 

During the design and implementation phases several 
problems commonly arise as a Student Outcome Model is applied. 
These problems can include: 

• Student objectives are not clearly defined and in 
those instances where they are the criteria for 
achieving the objectives are not specified. In 
other cases objectives may be unrealistically high 
or conservatively low. 

• Instruments used to assess progress often lack 
reliability (i.e., do they give the same results each 
time they are administered) and validity (i.e., do 
they assess what is being conducted) , These are 
often selected by individuals not familiar with the 
program or may be developed specifically for use in 
the program without any field testing. In other 
instances the conditions and procedures followed 
during the test administration differ from those 
described in a test manual. 

• . The analysis and use of test results are often 

inappropriate. A recent USOE publication titled 
A Practical Guide for Measuring Project Impact on 
Student Achiev ement (1975) identified eleven hazards 
in evaluating proje'cts with student achievement as 
a major objective. Some of the most common "misuses" 
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are using such nationally normed tests to assess an 
individual student ' s performance , administering the 
test over a time period different from that used for 
national norming, and the use of grade equivalent 
units rather than standard scores or raw scores . 

B. The Program Process Model 

The program process model is designed to determine the 
degree to which a project meets all of its major and minor 
objectives and/or to determine the degree to which certain processes 
are associated with student outcomes or other output variables. 

In the first instance, the Process Model is designed to 
evaluate the on-^going implementation of a project and the specific 
task and activities which are conducted by project staff as well 
as others. Depending upon the nature of the project, the major 
process objectives may include assessment of events (i.e. , whether 
a specific task was conducted on time) or activities (i.e., the use 
of resources to meet an event) . In this Model the evaluation 
activities are very similar to the monitoring activities as 
described earlier in Chapter III. A prerequisite for an effective 
application of this model is a well documented project plan with 
specified tasks, activities, and events and individual assignments 
and responsibilities . Moreover , criteria for task completion 
must be specified. 

The second function is usually conducted in conjunction 
with the Student Outcome Model since one is attempting to identify 
those specific processes which may be associated with (if not 
contribute to) the output variables. Rather than monitoring the 
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processes on an on--going basis the nature of the evaluation design 
may call for data collection of these processes through interviews, 
surveys, and/or observation of experimental effort or demonstration 
programs or pilot tests of certain techniques. The type of 
analysis which attempt to relate process variables (e.g., number 
of hours per day a student received a specific type of instruction) 
to outcome measures can be rather complex and sophisticated. The 
application of statistical techniques is used to draw inferences 
about the generalizeability of the results. 

While the problems in implementing the model for the 
first purpose are essentially those associated with project 
monitoring, the nature and degree of probleiBs associated with the 
latter function -are rather numerous including: 

• The overall design may not be comprehensive enough 
to capture all of the data on the most significant 
process variables thus resulting in limited or 
confusing findings . 

• Instruments used to gather data may be limited in 
terms of reliability and validity thus limiting the 
findings also . 

• The number of process variables analyzed may be so 
large compared with the sample or number of 
observations made txhat a number of findings may occur 
though as a result of chance. 

• And in those instances where process variables are 
significantly related (as determined by statistical 
applications) the ability to determine the degree 
to which a specific variable contributed to the 
student outcome may be severely limited. 
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The Policy Evaluation Model 

While the above two generic models could be classified 
as disciplinary research and evaluation models, a third generic 
model, the Policy Evaluation Model, is, as its name implies, 
designed to provide information useful for policy formulation. 
A difference in degree rather than kind, this model often involves 
many of the characteristics of the Process Model. While the Policy 
Model is sometimes applied at the district level, for the most 
part it is applied at the state and/or federal level to assess 
the results and effects of existing policies or to pilot test new 
programs. The general types of designs would include: a) a 
"planned variation" axperiment where two or three treatments, 
which are designed to differ by specific dimensions, are pilot 
tested in a number of districts; b) large scale surveys which 
establish the status of existing programs and their impact; and 
c) evaluations of exemplary programs funded under a specific 
legislative title in an attempt to determine which processes are 
associated with successful programs. 

In any of the above evaluations^ there exists a number of 
common problem areas from the perspective of the program director: 

• Areas of conflict arise between the design of the 
"imported" program and the district. In most instances 
criteria used for assessing program success will differ 
in' kind if not in priority ranking, between federal, 

state agency, and the district. The rationale for involve- 
ment may differ significantly. 

• The short term involvement in a pilot or experimental 
effort often results in staffing and continuity problems 
for the district and in certain instances public 
relations problems. i 
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• The inflexibility of the superimposed evaluation design 
may require a degree of inflexibility which often 
conflicts with the operational ease of administering 
and implementing the program at the district level. 

• And last, because of the problems noted above as perceived 
by the district, those confronted by the federal sponsor 
are often so major that either the evaluation design is 
erroded or the findings which result, in reality, reflect 
the ability of the agency to conduct the experiment 
rather than the technique being tested. 

Very briefly then, increased priority will continue to be 

placed upon evaluation; rather than becoming more simple the 

proliferation of evaluation approaches and models will increasingly 

become more con.plex as attempts are made to determine causal 

relationships between program processes and outcomes or at least 

attempt to explain the success or failure of programs; and in turn 

those responsible for the operations of programs at the district 

level will increasingly have to become more familiar with, if not 

experienced in, the use of evaluation techniques. 



GOALS OF EVALUATION 

Earlier we described the two major purposes of evaluation. 
Here, we describe the specific goals of any evaluation task which 
necessarily differ among the various types of programs in education. 

For the purposes below we have identified three general 
categories of special projects. 

• Planning and development projects whose end product is 
a plan or a product (e.g., a curriculum) on which 
subsequent funding will be based; 

• Experimental and demonstration projects whose primary 
purpose is to test and evaluate experiments which assess 
the use and value of materials, techniques, methods. 
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• Operational projects whose primary purpose is to carry 
out activities aimed at accomplishing specific 
educational obj ectives . 

While each of the above projects may involve phases of the 

other (e.g., an operational project may be preceded by a planning 

project) , we have identified a number of evaluation goals of the 

three categories to illustrate potential differences, as described 

in Figure 4 . 



FIGURE 4: GOALS OF EVALUATION 


Planning 
and 

Development Projects 


Experimental 
and 

Denonstration Projects 


Operational Projects 


To fiolfill funding agency 
and other reporting 
requiran[>eiits . 

To improve the organiza- 
tion and iirplementation 
of subsequent planning/ 
developiental projects 

To maximize the adequacy 
and etficiency of sub- 
sequent tests and 
iiTplewentations . 

To irrprove the selection 
of subsequent planning/ 
developsnental projects. 


To fulfill funding agency 
and other reporting 
requirenents. 

To iirprove the design anS 
implementation of sub- 
sequent E & D projects. 

To maximize the chances 
of incorporation of the 
project in the system if 
it is successful. 

To maxiinize replicability 
if project is successful* 


To fulfill funding agency 
and other reporting 
requirements . 

To irtprove the design and 
iirplementation of sub- 
sequent operational 
projects. 

To maximize the effective 
application of resources 
in relation to continuing 
or changing needs. 
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EVALUATION FUNCTIONS 

Regardless of the specific goals of an evaluation design, 
there exists several functions to be provided by the person or 
persons responsible for conducting the evaluation'. These functions 
are suimnarized below. However, the specific sequences may vary in 
light of several factors (i.e., whether the evaluation will be 
conducted internally or by an outside group) . It is critical to 
note that specific evaluation functions need to be undertaken 
during the planning, preparation, and operational phases of a 
project. 

A. Planning Phase 

As noted earlier, one of the first evaluation functions 
is to conduct a needs assessment. Whether this function is filled 
by the chief planner in preparation of a proposal or by an 
evaluator is less important than the recognition that this function 
be fulfilled. As the program planning to develop a project to 
meet specified objectives is undertaken, several additional 
evaluation functions are required. 

First, project information must be delineated. A number 
of factors must be considered at this time including: 

• Are the objectives of the project clearly defined 
designating a target group, a behavior or condition 
to be brought about, the means for measuring 
performance and the conditions under which performance 
will be observed? 

• Are the processes for accomplishing these objectives 
defined regarding who, when, how often, how much, how 
long, as well as intended results and criteria for 
completion? 
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a Have all objectives been reviewed and approved by 
participants? 

• Has the target population been specified? 

• Have the major milestone dates for project component 
activities been established? 

• Does a monitoring system exist or is one being 
planned? 

And, upon delineating project information, a key 
evaluation function is to assess the needs of the users of "information 
emanating from the evaluation of the project — specifically, who 
needs what information, for what purposes, when, and in what format • 
The users of evaluation information usually range from the funding 
agency (in the vast majority of cases) to individual teachers who 
may want to use information to improve the program. Since it is 
impossible to provide all information to all potential users, due 
to time and cost constraints, it will be necessary to negotiate 
priorities. Specific factors to be considered in determining user 
needs include: 

• Minimum information and reporting requirements 
specified by the funding agency; 

• The kinds of information sought or questions ;raised 
by various types of decision makers, their priorities, 
and when the information is needed; 

• The possibility of conflict between (if not 
differences, in priority ranking) of criteria used to 
assess the project and those criteria used by decision 
makers ; 

• On-going information user needs prior to completion 
of the project. 

The last evaluation function usually conducted during the 

planning phase of an overall project is the preparation of an 
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evaluation design. For each project objective there should be an 
evaluation objective. Similarly, project staff should agree upon 
the evaluation objectives. There should be an implementation plan 
for the evaluation design similar to the implementation plan for 
the overall project. In developing the evalBation design several 
additional factors must be taken into account, including: 

• What is the primary unit of ar.alysis — the 
individual student, the classroom, the school, or 
other? 

• Are instruments proposed to assess pro j ect success 
valid, reliable, and economically feasible to 
administer? 

• What, if any, sampling should be considered 
regarding data collection? Can it be justified? 
Will sampling a limited number of persons affect 
those not sampled in a partic^izdar way? 

Once the evaluation design has been developed in a 

preliminary way, two additional questions shtjuld be raised: 

• Is the design technically sufficient, practical in 
terms of utility, and efficieiit in terms of cost and 
other resources necessary to implement it? 

• Is there general agreement between those associated 
with th^ project that the evaluation will be 
relevant , important , credible^ and comprehensive? 

B. Prep aration Phase 

Once the evaluation design is completed and usually 

submitted as part of a proposal to the sponsoring agency and the 

project is approved, then a number 'of steps are required to be 

undertaken by the project director or the individual responsible for 

evaluation. Some common problems and revisions in difficult projects 

include: 
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0 The need to reduce the number of evaluation 

objectives and to modify them due to changes of 
project objectives (typically proposals are overly 
ambitious and unrealistic in the number of 
accomplishments that they propose to make) . 

0 Revising the evaluation budget due to overall budget 
reductions • 

0 Revision or selection of new instruments based on 
discussions and desires of the funding agency. 

Second, usually at this stage of the project a decision 

has to be made whether or not the evaluation is to be conducted 

internally or whether it should be contracted out to an independent 

third party. In a large number of programs funded under Federal and 

state legislative titles provisions require that a third party 

evaluation be conducted. If this is the case, the project director 

is responsible for drafting a Request for Proposal based upon the 

evaluation design and submitting it to a number of potential bidders 

who in turn will respond with proposals. In other instances, an 

individual may have already been designated, in which case an 

evaluation contract is developed. It is critical that the 

specifications for the evaluation be developed by the project staff 

rather than asking the evaluator to develop his own design of 

evaluation. Several factors can create subsequent problems 

including: 

0 The lack of early involvement in the planning phase 
by the individual responsible for evaluation, a 
potential problem when evaluations are contracted 
out ; 

0 Lack of coordination and division of labor between 
project staff and the evaluator which usually is a 
direct result of an inadequate overall management plan 
and the lack of specificity in the evaluation contract, 
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Third, a critical evaluation function is the obtaining 
and/or development of instruments to be used to collect data. In 
addition to ensuring that instruments are appropriate, relevant, 
valid, and reliable, other factors should be taken into account, 
including: 

• The cost of obtaining and implementing the instrument, 
since there exists hidden costs in many cases (e.g. , 
requirement to use the publisher's computer to 

score the results) . 

• The human resources and other costs to implement 
the instrument may be underestimated. 

Fourth, depending upon the nature of the project and the 

evaluation design, the most critical is the training and scheduling 

of data collectors. Wherever possible, it is desirable that 

individuals a.lready trained in the administration of specific 

instruments be used. However, this ideal situation does not always 

exist. A carefully designed and conducted training program perhaps 

is the most critical aspect of ensuring quality control over the 

overall data collection process. In many instances, especially 

where observation instruments are used, it is desirable to actually 

conduct observations on a pilot basis to ensure that there exists 

general agreement among observers, often referred to as interrater 

reliability. During training sessions it is often desirable to 

obtain information regarding individual data collector's schedules, 

since they. may not be fulltime employees of the school and develop 

a tentative schedule. 

68 



-52- 



The result of the preparation phase as far as evaluation 
activities are concerned is an evaluation apparatus to implement 
the overall evaluation design. 
C • Implementation Phase 

The major task during the implementation phase of an 
evaluation design are data collection, data processing, data 
analysis and reporting, and the development of recommendations 
including those for program improvement. 

First, during the data collection phase a number of 
problems which could potentially erode the evaluation design often 
occur including: 

• Logistical and rescheduling problems due to the 
.vinavailability Qf staff thus requiring contingeney 

plans and opportunities for rescheduling; and 

o Collection of incomplete data requiring personal or 
other follow-up since many evalu^ion techniques 
are based upon the assumption that data will be 
complete. 

Second, once the data are collected a major evaluation 
function is the reduction and processing of data. In processing 
data a number of considerations must be taken into account: 

• The need to reduce the number of times a particular 
piece of data is manipulated, thereby reducing the 
probability of human errors ; 

• The opportunity to have the data collectors reduce 
data as much as possible, a function which requires 
extreme quality control , however ; 

• Whether or not to use automated data processing 
which requires, once again, high quality control 
yet may be necessary due to the nature of the 
analysis to be used; 
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• Ensuring that the data are in an easily retrievable 
and accessible format for those conducting the 
analysis. 

Third, the primary evaluation function involved here is 
the analysis and reporting of findings to decision makers in a 
useable format on a timely basis. As noted earlier, a number of 
analytical techniques can be used; moreover, the opportunity to 
use additional techniques is often afforded when the data are 
processed through use of automated mechanisms and/or computers. 
Findings may be presented in a rather raw form; in most instances, 
however, especially when sampling is used, the application of., 
statistical techniques is required to draw inferences regarding 
the generalizeability of the findings and the "statistical 
significance" of the results. Once the data are analyzed a report 
or a number of reports usually are developed, taking into account: 

• The specific amount o-^ time users of ^ le information 
and decision makers will have to make judgements 
thereby requiring executive summaries; 

• Whether or not to develop a number of separate and 
differently formatted summaries for various types 
of audiences such as the general public and/or 
individual teachers . 

And last, depending upon the nature of the project, a 

major evaluation function is to provide feedback and follow-up to 

various users of the information. Or in the event that the specific 

project is to have a subsequent phase, the evaluator should be 

involved in the planning for that phase v/here possible to ensure 

some degree of continuity and program formulation based upon 

previous findings. ^ 
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CHAPTER VI 

AN EVALUATION MODEL FOR COOPERATIVE EDUCATION 

Cooperative education needs an evaluation model which can be 
molded to fit the needs of a variety of program types and program 
settings. The model must be equally useful for distributive 
education, office occupations, industrial and trade occupations 
programs and other subject areas and combinations of subject 
areas; for large cities and rural areas; for secondary and post- 
secondary schools; for standard classes and classes for students 
V7ith special needs. The model must ^.llow for input on evaluation 
questions from all concerned groups students, parents, teachers 
administrators, employers, labor representatives, community 
groups, and advisory councils. It must be sufficiently straight- 
forward and practical that cooperative education program directors 
with limited resources at their disposal will find it realistic 
and useful. Yet, it must also provide for the use of sufficiently 
sophisticated techniques so that evaluation questions concerning 
the progress and outcome of instruction, and the relationehip of 
program components to outcomes, can be answered." 

The model described in the following pages, referred to as 
the Cooperative Education Evaluation Model, ineets these require- 
ments. This model, based on a general evaluation model developed 
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by Fink and Kosecoff-^, has three features that distinguish it 

from ether models and make it especially appropriate to cooperative 

education programs : 

1. Responsiveness to a program's needs — the evaluation 
model has been specifically desigiaed so that it can be 
molded to fit the requirements of every program no 
matter how unique they are. There are no mandatory 
evaluation phases or stages ^ and bo present evaluation 
designs or information analysis methods • Instead, the 
model advocates the completion of a series of activities 
that are coordinated to answer clients' evaluation 
questions. Thus, with this model, the shape of an 
evaluation is entirely dependent mpon the purposes and 
nature of the program and is nevex imposed on it. 

2. Provisions for checks and balances — because of the 
importance of clients' evaluation questions to this 
model, each major category of evaluation activity 
begins and ends with reference to previous activities 
and to the need for consultation -uith the client to 
ensure that his/her needs 'are being met. 

3. Action and practical orientation — the model has been 
developed so that the evaluation ean provide timely, 
relevant, and accurate information that can readily be 
used. This is done by providing the framework for 
including the client in the formation of the evaluation 
and the monitoring of its progress and quality. 

Evaluation is defined here as "a set of procedures used to 

appraise a program's merit and to provide information about the 

nature and quality of a program's goals, outcomes, impact, and 

costs". There are two contexts in which evaluations can be 

conducted using the Cooperative Education Evaluation Model. In 

the first context, an evaluation is conducted to improve a 

program and the evaluation's clients are typically the program's 
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organizers and staff. In the second context:/ an evaluation is 
conducted to certify a program; here, the evaluation's clients 
are typically the program's sponsors. 

The context for an evaluation is determined by the inforxn't- 
tion needs of the individuals and agencies who must use the 
evaluation information. An evaluation is performed in an 
improvement context when the evaluation's clients are concerned 
with finding out precisely where a change woold make the program 
better. Usually, the organizers of a still—developing cooperative 
education program require this kind of infoirmation so that they 
can modify and improve the program. On the other hand, an 
evaluation is conducted in a certification context when the 
evaluation's clients are particularly concerned with determining 
the extent to which the program's overall quality can be guaranteed 
Those individuals who sponsored prograiri development, or who are 
interested in using the program, require this kind of information 
about a completed program's outcomes, impact, and costs. In 
addition, in a certification context, the evaluator frequently 
assumes a more global and independent perspective than in an 
improvement context. 

The evaluation activities are organized into six major 
categories : 

1 . Selecting evaluation qui stions , 

2. Organizing information collect:'. on jr 

3. Collecting information, 
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4, Planning and implementing data analysis, 

5, Reporting information, 

6, Managing evaluation activities. 

Chapter VIII contains an explication of each of these six 
categories of evaluation activities including an overview of 
each, a list of criteria for implementation, and a list of 
cautions regarding common problems encountered. 

Chapter IX contains an example of instructional materials 
based on Component 1; Selecting Evaluation Questions. 
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CHAPTER VII 
Planning/Management 

A. Guidelines for Content on Planning Management 

B. Checklist on Planning Management 
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A. Guidelines for Content on Planning; and Management 

In previous chapter3 we identified general purposes and 
techniques for planning and managing special educational programs 
and projects and discussed problems specific to the planning and 
management of cooperative education programs. This section 
contains guddelines for the content of training materials for 
cooperative education program directors on the topics of planning 
and management "based on the concepts contained in those earlier 
chapteTs. The plaijaing-management task is seen as a continuum, 
v/hich is divided for convenience into the following phases: 
planning, preparation, implementation, and completion. ?or each 
of these phases there is a brief discussion of considerations 
which contribute to success and a list of common problems. A 
checklist- covering all of the phases is included at the end of 
the section. 

1. Planning; jghase 

Good planning is probably the most important single feature 
of a successful cooperative education program. If the goals and 
objectives have been appropriately selected and defined and the 
management plan has been v/ell constructed, the program can proceed 
smoothly. If these first crucial tasks are not properly handled, 
it is extremely difficult to make up for the lack later in the 
course of the program. 

Considerations in Successful Plaruing 
The planner shouDd: 
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1. Gather information on desired goals and objectives from 
all interested groups including students, parents, teachers, admin- 
istrators, employers, labor representatives, community groups and 
advisory councils. Involve all interested groups, and particularly 
employers who will be providing the work stations., in the planning 
from the start. 

2. Think through the theoretical and practical implications 
of each, selected goal or objective, particularly those related to 
job placement, 

3. Derive and clearly define a set of student outcome objec- 
tives which can .reasonably be expected to be accomplished with the 
time and resources available. Use best possible information on labor 
market projections and availability of good vrork stati' ^ 

4. Por each student outcome objective define: 

a. what tasks and activities v/ill be undertaken to 
accomplish the objective 

b. what criteria v/ill be used for judging to v;hat 
extent the objective has been achieved^ 

5. Define program process objectives or milestones and the 
time by which each will be accomplished. 

6. Define the . responsibilities of each staff member. Define 
the responsibilities of all participants. 

7. Using the above information, write a management plan, 
including a flow chart, v/hich shows time/task relationships and 
responsible personnel. 

8. Design and build in the program evaluation from the 
beginning, making it an integra.1 part of the management plan. 

9. Review the ...plan with appropriate staff and other interested 
parties. 77 
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Connnon Planning Problems 

!• It may be difficult to achieve concensus on student 
outcome objectives because of the differing vievrpoints of the 
varied interested groups, 

2. Some objectives which are desired may be difficult to_ 
measure, complicating the development of criteria by which to 
judge their accomplishment, 

5. Budget constraints, lack of qualified staff, or lack of 
sufficient work experience stations may force a scaling-down of 
objectives and activities. 

4. It may be difficult to achieve consensus on the program 
process objectives and the management plan because of differing 
vievrpoints on the part of key staff. 

2. Preparation phase 

After formal plans have been made, the program director must 
see to it that all components of the program arejLn place and 
ready to go. This is an important phase in itself and should not 
be short-changed in terms of time and resoiJirces allowed for its 
accomplishment • 

Conside rations in Successful P rep aration for the Program 

The cooperative education program director should be sure 
that the following tasks have been accomplished before beginning 
the actual implementation of the program: 

. 1. Confirm commitments from employers and all others 
involved in the program* If modifications are necessary, be sure 
proper actions have been taken» 
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2. Issue final schedules for students, staff, employers* 
5. Confirm that agreements with employers and students are in 
proper order. Confirm that all pa:,ties understand the training 
agreements and that all questions of privacy rules have been dealt 
with and written permission for access to student data obtained. 

4. Brief the advisory council and other interested parties 
on program plans. 

5. Confirm that a system for collecting data needed for 
program management and evaluation is in place. 

Common Problems in. the Preparation Phase 

1. Work that the director assumed was done has not been done 
or has not been adequately done# 

2. Last minute modifications have to be made because of 
changed circumstances. 

^. Program implementat ion 

If the planning and preparation phases have been carried out 
correctly, then the director's job in implementing the program is 
simplified considerably. The director's major concern should be 
to assi5.re that activities are going as planned and to make 
modifications where changed circiomstances or new information 
indicates that they should be made. 

Considerations for Successful Im_p_lementation 

The cooperative education program director should continually 
monitor the implementation of the program plan throughout the 
course of the year, including the following tasks: 

1. Check actual accomplishment of tasks against the time/task 
outline in the management plan or flow chart. 



2* Confirm that all needed data is being collected in usable 
form for management and evaluation purposes, including information 
on the usefulness of each work station; assessment of gaps in each 
student ^s readiness for employment; records of the development of 
each student's skills; information on the extent of coordination 
between v/ork-site experiences and in-^class instruction; placement 
and follow-up data* 

3. Determine that information on problems within the program 
is reaching the director or other appropriate staff member and 
that proper corrective action is being taken* 

4. Confirm that information about the progress of the program 
is reaching all appropriate parties including parents, advisory 
councils, community groups and others. 

5. Confirm that the evaluation is proceeding as planned and 
that interim evaluation information is being used, where appropriate, 
to modify the program. For example, evaluation of a work site may 
have yielded information that the site is inappropriate for students. 
Those students then must be shifted to an alternative site. 

Common Problems in Implementation 

1. Delays occur in certain parts of the program. 

2. Unjjredictable problems force modification of the program 
plan in mid-year. 

3. The flov/ of information from program participants to the 
director is inhibited, complicating the director task in dealing 
with problems. 

, 4. P roj ^ram completion 

At the end of the year a number of tasks must be done to assure 
proper documentation of the program, to lay the groimdwork for the 
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following year's program, and, in those cases where outside fimding 
sources require it, to provide a final repori to the f\znding source. 

Considerations for Program Completion 

The cooperative education program director should: 

1. Confirm that all tlie information necessary to document 
the program has been gathered and is properly filed. 

2. Confirm that the evaluation is proceeding as planned. 

3. Assure that preparation for a follo^up of graduates has 
been accomplished. 

4. Confirm that the final report, if re^quired, is being 
drafted and will be supplied to the sponsoring agency on time and 
in proper form. 

5. Assure that preparations have been iiade for the conduct 
of a fiscal audit, if appropriate, including: 

a) docTomentation of changes and approved contract 
modifications 

b) updating and indexing files 

c) storing for a possible audit in the future. 

6. Assure that proper steps have been taken to inform advisory 
coimcils, parents, employers and community graups about program 
successes and future plans. 

7. Begin planning for next year's program. 

8. If the program is to be terminated, check that appropriate 
measures have been taken to terminate or roa:ssign staff and to 
dispose of materials and equipment. 

Common Problems at Prof^ram Completion 

1. Information is not in proper order. 

2. There is uncertainty about the future of the program because 
of funding or staffing problems. 81 



B. Checklist for Planning and Managing 
Cooperative Education Programs 



The follov/ing checklist contains v^ucstions that can seirve 
as an aid in assessing planning and management procedures for 
cooperative education programs • The questions in the checklist 
are organized according to four phases or program operation: 
planning, preparation, implementation and completion. It should 
be noted that the questions may he modified, when necessary, to 
suit your program's unique needs and that questions may be added 
or deleted. 



Response 

Question Yes No 

A. Planning 

1. Has information been gathered on desired 
goals and objectives from all interested 
groups including students, parents, 
teachers , administrators , employers , 
labor representatives, community 

groups and advisory coimcils? 

2. Have tl.eoretical and practical implica- 
tions been thought through concerning 
each selected goal or objective, 
particularly those related to job 

placement? ^ 

3. Have student outcome objectives been 
derived and clearly defined v/hich can 
reasonably be expected to be accomplished 
with the time and resources available? 



4. Have student outcome objectives been 
defined fors 

a) what tasks and activities will 

be undertaken to accomplish the 
objective 

b) what criteria will be used for 
judging to v/hat extent the 
objective has been achieved? 
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Qv&stlon 



Response 
Yes No 



5* Have program process objectives or 
Tiilestones teen defined and the time 
by which each will be accomplished? 

6, Have the responsibilities of each 
staff member been defined? 

7. Using the above information, has a 
management plan been written including 
a flow chart, which shows time/task 
relationships and responsible personnel? 

8, Has the program evaluation been designed 
and built from the beginning, making it 

. an integral part of the management plan? 

9. Has the plaji been reviewed with appro- 
priate staff and other interested 
parties? 

10, Has the program budget been reviewed to 
be sure that it is realistic in terms 
of activities to be xmdertaken end 
availability of resources? 



B, Preparation 

1. Have commitments been confirmed from 
employers and all others involved in 
the program? If modifications are 
necessary, have proper actions been 
taken? 

2. Have final schedules been issued for 
students, staff, employers? 

3. Has it been confirmed that agreements 
with employers and students are in proper 
order? ' 

4. Have the advisory council and other 
interested parties been briefed on 
program plans? 

5* Has it been confirmed that a system 

for collecting data needed for program 
management and evaluation is in place? 
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Question 



C. Prograjn Implementation 



Yes. No 



1. Has actual accomplishment of tasks 
been checked against the time/task 
outline in the management plan or 
flow chart? 

2. Has it been confirmed that all needed 
data is being collected in usable f^crm 
for management and evaluation purposes? 

3. Has it been determined that information 
on problems within the program is 
reaching the director or other appru.-- 
priate staff member and that proper 
corrective action is being taken? 

4. Has it been confirmed that information 
about the progress of the program is 
reaching all appropriate parties 
including parents, advisory councils, 
coramtmity groups and others? 

5. Has it been confirmed that the evaluna-- 
tion is proceeding as planned and tliiat 
interim evaluation information is teing 
used, where appropriate,., to, modify the 
prograjn? 



D. Program Completion 

1. Has it been confirmed that all the 
information necessary to document tie 
program has been gathered and is 
properly filed? 

2. Has it been confirmed that the 
evaluation is proceeding as planned? 

3. Is it assured that ^reparations for a 
follov/-up of graduates has been 
accomplishes? 

4. Has it been confirmed that the final 
report, if required, is being drafted 
and will be supplied to the sponsoring 
agency on time and in proper form? 
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Question 



Response 
Yes No 



5. Is it assured that preparations have 
been made for the conduct of a fiscal 
audit, if appropriate, including: 

a) dociHuentation of changes and 

approved contract modifications 
"b) updating and indexing files 
c) storing for a possible audit in 
the future? 

6. Is it assured that proper steps have 
been taken to inform advisory councils, 
parents, employers and community groups 
about program successes and future 
plans? 

?• Has planning begun for next year's 
program? 

8. If the program is to be terminated, 

has a check been made that appropriate 
measures have been taken to terminate 
or reassign staff and to dispose of 
materials and equipment? 
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CHAPTER VIII 
Evaluation Model CompoiierLts 

A, Selection of Evaluation Questions 
Organizing Information Collection 
Collecting Information 

D. Plcuining and Implementing Data Analysis?. 
Reporting Information 

E. Managing Evaluation Activities 
G. Checklist 
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A. SELECTION OP EVALUATION QUESTIONS 

Introduction 

An effective evaluation results in timely and "believable 
information that is useful in improving or certifying a program. 
To guarantee an evaluation's effectiveness, tiie evaluator must 
pose questions that are responLive to the needs of all concerned 
individuals. In formulatin, questions, the evaluator must 

review the program's goals ar.d \ctivities and ascertain the kinds 
of information that v/ill acceptable as evidence of program 
success . 

Considerations Involved in Selectin.g: Evaluation Questions 
To ensure a credible evaluation, the evaluator must: 
!• Reviev/ the program's goals and activities. 

V/hen reviev/ing a program's goals and activities, the 
evaluator should become familiar v/ith the interests a-rd concerrs 
of all groups v/ho have a stake in the program and its t;valuaxioi> ; 
e.g., student, parents, employees, teacher, adminis^trator, 
advisory committee members, and/or funding source. 

2. Be responsive to the types of infonmtion that will be 
convincing as evidence of the program's success. 

There are many d:, erent ways to prove that a program has 
been successful; e.g«, that its goals have been achieved, that it 
was managed well, or that it had no negative effects. Some ways 
include records of successful placements of students in jobs, 
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successful performance of students on test of skills and testimony 
from program graduates ^ employers and parents. 

Pose specific questions that the evaluation's audiences 
want answered. 

Evaluation questions can take the following forms; 

• To what extent were the program's goals achieved? 

• V/ere the program's activities implemented as planned? 

« How effective were these activities in achieving 
the goals? 

• For which groups was the program most/least successful? 

• What did the program cost? 

• How v/ell was the program managed? 

• Hov/ did external and internal social and political 
forces influence the program* s development and 
impact? 

• VHiat social and political effects did the program 
have on the environment in which it v/as implemented? 

4. Make sure that those participating in the evaluation 

understand the procedures and products of the evaluation. 

The evaluator must make sure that appropriate participants 

understand v/hat an evaluation is, the reasons for conducting the 

particular evaluation, how evaluktion information will he used, 

and that needed releases for use of informat5.on have been obtained 

from students cr their parents v/here necessary. 

Problems in Selecting; Evaluation Questions 

1, The evaluator may have difficulty in obtaining the 
cooperation of participants. 
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2. Access to the program's documents or staff for evaluation 
can be limited or not possible because of privacy regulations, 

5, The program's goals may have been unclearly or not 
meas^irably stated, and/or the program's activities may be 
imprecisely described. 

4, Different interest groups might be unwilling or unable 

to agree on the evaluation questions and on what will be acceptable 
as evidence of the program's success, 

5, The evaluation questions might not be on target because: 

• there are too many/too few 

• they do not le: d themselves to adequate answers, given 
time and money that are available. 
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B. ORGANIZING INFORMATION COLLSCTION 
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Introduction 

Planning an evaluation's information collection activities 
involves taking into account the evaluation q.uestions, the infor- 
mation collection techniques, and the design strategy used to 
group and sample participants and to structure the data analysis. 

Considerations Involved in Organization for Information Collection 

1. Techniques used to collect evaluation information. There 
are a variety of techniques that can he used to collect evaluation 
information including^ interviews, questionnaires, rating scales, 
observations, record reviev/s, and achievement tests. Each has 
advantages and disadvantages, and the evaluator must determine 
which will yield the most reliable and valid information, given 
the inevitable constraints of time and money, 

2. pesiffa strategies used to group and sample participants. 
Frequently used design strategies for cooperative education programs 
should include case study designs; time series designs that compare 
the project population's gains with previous years* scores j and 
comparison group designs that include control groups, comparisons 

of gains over time v/ith national norms, or comparison of gains 
when yoimg people have a high level of involvement with the 
project activities with the gains achieved "vifeen they have a low 
level of involvement, 

90 



-74- 



3. Samplin/:;, Sampling guides the selection of persons to be 
used in the evaluation and the assigrjuent of these persons to 
groups. The evaluator must determine whether some or all eligible 
studentSj teachers, employers, administrators, or advisory committee 
members v/ill be included in the evaluatrLon, and v^hether or net they 
will participate in xhe nev; or traditional program. 

Problems in Orp;ani2in/?; Information Collection 

1. The schedules of a program's participants; e.g., employers 
and teachers; do not always coincide v/ith the evaluation's 
schedule. 

2. The most desired information collection techniques may 
not be the most reliable or valid, and they may be the most 
expensive* 

3. The evaluator might prefer a particular design, but be 
unable to implement it because comparison groups are unavailable, 
data cannot be identified from previous years, etc. 

4. Difficulties arise in obtaining information about eligible 
participants because of privacy regulations, inability to obtain 
participant cooperation, and mobility. 
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C. COLLECTING INPORMATION 

Introduction 

Collecting evaluation information is a large and complex 
task that has a direct "bearing on the quality of the resulting 
evaluation information. Poor information collection instruments 
can yield invalid and possibly false information. 

Considerations Involved ;i Successful Information Collection 

!• Selecting, adar)tinR, or develoDin^ instruments. The 
first step in collecting information for the evaluation of coopera-- 
tive education programs involves the evaluator in selecting, 
adapting, or developing reliable and valid instruments to 
measure the effects of the program. 

2. Hirin/? and Training Information Collectors. Information 
Collectors can be selected from the program staff itself, profes- 
sional organizations, and the commimity. Once hired, collectors 
must undergo rigorous training. 

3 • Pilot testing information co ll ection instruments and 
procedures . Before using information collection instruments and 
procedures, they should be pilot tested to help answer questions 
like: 

. Hov/ accurate is the information obtained v/ith the 
instruments (validity)? 

• How consistent is the information obtained by the 
instruments (reliability)? 



4, Conducting: information collection* Infurmacion collection 
oan mean obtaining ^'clearance," from agencies like the tJ, S, Office 
of Manageinent and Budget (Ol-IB) and informing participants of the 
purpose a.nd nature of their cooperation. 

5* Monitorinp; information collection* Information collection 
should be supervised to ensure that activities are being conducted 
correctly and that all necessary data are being gathered. 

Problems in .Information Collection 

1. Difficulties arise in identifying validated instruments, 
and development is expensive or time-consuming* 

2. The best information collectors are unavailable or too 
expensive to hire or train* 

3. A pilot test can be too small or inadequately performed 
to provide reliable information. 

4. It is sometimes difficult to allot the necessary amount 
of time for clearance (often several months) of nev/ly-developed 
instruments, and validated ones are not available* 

5. Once infor \ eligible participants may withdrav/ from 
the evaluation* 

6* Difficulties arise in obtaining the cooperation of 
participants* 

7, Difficulties arise in collecting information from partici- 
pants v/ho move away or lose interest in the program after completing 
their formal participation in it* 
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PLANNING AND IMPLEMENTING LATA ANALYSIS 

Ii itro duct ion 

The analysis of evaluation information is the process by 
which all the data obtained during the various information collec- 
tion activities are suimnarized and synthes: -ed to produce aasv;ers 
to evaluation questions. Analysis methods range from the 
statistics-based techniques used by psychologists and sociologists 
to the scholarship-based tecl-miques often used by historians and 
anthropologists. All attempt to describe evaluation information 
in the form jf tallies or measures of variation, and to explain 
evaluation information by identifying patterns and trends in 
events. 

Considerations Involved in Successfull y Plan nin/s; and Implementing- 
Lata Analysis 

1. Planning , the data analysis. -Analysis activities must be 
carefully planned to be technically appropriate, responsive to the 
evaluation questions, and, in turn, compatible v;ith the design 
strategy and information collection techniques. The selection of 
specific analysis raethods v/ill usually be influenced by the 
evaluator^s training and background and the. resources available for 
the evaluation. 

2. Conducting data analysis activities . Completion of 
analysis activities must include more than just uhe actual perfor- 
mance of the analysis. It must also involve: 
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' I'-educing the evaluation information to usable form 

• pilot testing the information analysis activities 

• conducting the analyse 

• archiving the evaluation inf oraation. 

Problems in Planning and Implementing Data Analysis 

1. The evaluation questions are un.clear, and it is difficult 
to tailor the analyses to them, 

2, The evaluator's personal training or background influences 
him or her to accept an expensive or otherv/ise inappropriate 
analysis method. 

5. The design strategy has been improperly selected or poorly 
implemented, or the information collection techriques are unreliable 
or invalid yielding uninterpretable results. 

4» xx)o much informati-on is collected^ or it is badly reduced 
delaying the performance of the analyst's tasks. 

5. 1 ilot testing can reveal the need to revise some or all 
analysis teclmiques because the v/i'ong data v/ere collected, they 
are insufficient to provide ansv/ers to the evaluation questions, 
etc « 



ERIC 



95 



-79- 



E. REPORTING INFORMATION 



Introduction 

An evaluation report consists of the answers to some or all 
of the evaluation's questions and an explanation of the procedures 
used to derive the onsv/ers. The evaluation report, whether written 
or oral, informal or formal, is an official record of the evalua-- 
tion. It is through the report that the e valuator makes public 
his or her acti^.^ities and findings. Thus, it is essential that 
the evaluation's audience be given easy access to reports and that 
they be clearly vn?itten. 

Considerations Involved in Successfully Re-pci'ting Information 

To be credible, am evaluation report must be easily understood 
by all its readers or listeners, including parejxts, teachers, 
students, employers, advisory committee members, administrators 5, 
and funding agencies. In preparing the report, the evaluator 
should consider including the following: 

• an introduction to the evaluation including its backgromid, 
the evaluation questions, and limitations on the scope of 
the evaluation 

• the collection of evaluation information including the 
design, sampling, information collection techniques, 
limitations on the information collection activities 

• the methods used to analyze the data and their limitations 

• th^ evaluation findings including answers to each evaluation 

question, interpretations, recommendations, and limitations 
on the findings 

• mar.agement concerns like schedules and staff assignments, 
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The importance of each of these considerations will depend 
upon the nature of the program, the evaluation, and the purposes 
for which the evaluation report will "be used. 

Problems in Reporting Information 

1. Technical matters are sometimes difficult to translate 
into terms that all appropriate audiences; e.g., students, 
employers and funding agencies; can understand v/ithout oversim- 
plifying. 

2. It is hard to assign priorities to the information so 
that only the most important is emphasized. 

3« Reconstructing evaluation events can "be difficult and 
time-consinning. 

A. Because of the possibility that the evaluation's findings 
may be misinterpreted or taken out of context, the evaluator some-^ 
times feels (or is pressured to feel) the need to distort findings. 

5. Evaluation reports sometimes appear overly critical or 
too full of praise rather than providing a balanced view. 
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p. MANAGING EVALUATION ACTIVITIES 



Introduction 

Ability to manage and coordinate evaluation activities is 
essential, and at least some portion of the evaluator's time must 
be given to management. It is only through careful attention to 
schedules, tasks and budgeting that the evaluator can assure 
teachers, students, employers, parents, funding agencies, and 
advisory councils that they will get timely and usable answers 
to evaluation questions. 

C onsiderations Involved in Successful Manapjement 

1. Establishing; schedules. Evaluations are commissioned 
to be conducted within a given amount of time. To ensure the 
success of the effort, the evaluator must determine v/hen each 
evaluation activity will take place, the sequence of the activities, 
and how long each one will take. 

2. Assigning staff to activities. In order to assign staff 
to specific evaluation activities, the skills needed to perform 
each activity must be identified, so that the staff members with 
those skills can be assigned appropriately. 

3. Budgeting. To prepare an evaluation's budget, the 
evaluator must weigh what needs to be done against the amount of 
money that is likely to be available. Invariably, activities, 
time allocations, and staff assignments are juggled during the 
development of the budget. 
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Problems Involved in Successfully Managing an Evaluation 

1. There is never enough time or money to do the perfect 
evaluation! 

2. Trained staff may be difficult, to find. 



99 



-83- 



G. CHECKLIST 



The following checklist contains questions that can serve as 
the basis for evaluating the effectiveness of a cooperative 
, education program. The questions on the checklist are organized 
according to the six categories that comprise the Cooperative 
Education Model* It should be noted that the questions may be 
modified, when necessary, to suit your program''s unique needs, or 
that questions may be added or deleted. 

Response 

J Question Yes No 

1. Have the interests and concerns of the 
school employers, student-trainees, 
graduates, teachers, parents, advisory 
committee members and/or fimding agencies 

been reviewed? 

2. Have participants been asked to suggest 
the types of- evidence they will accept as 

proof of the program* s merit? 

5. Have evaluation questions been identified 
that relate to the program* s objectives, 
activities, costs and social, educational, 
and political impacts? 
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Response 

Question ' Yes No 

4* Have information collection techniq.ues 

been identified that will provide valid 

and reliable information about attitudes, 

perfoimance, skill or knowledge? ^ 

5. Has an evaluation design been formulated 
that will permit comparisons of the 
program's gains over time or between 
performance in the program and national 
norms or among degrees of student involve- 
ment in the program? 

6» Should sampling be used to select 

students, employers, parents and adminis- 
trators for participation in the evalua- 
tion, or to assign some or all to 

different groups? 

7. Can reliable and valid instruments like 
achievement and performance tests, 
observations, rating and ranlcing scales^ 
questionnaire survey forms, interview 
schedules, and record -review forms be 

purchased and used without modification? 

Adapted to meet the program's needs? ^ 

Specially developed for the program? ^ 

8. Are information collectors available for 

hiring and training? 
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Question 



9. Have provisions been niade to pilot test 
and validate information collection 
procedures arid instruments? 

10. Have "clearance" and privacy regulations 
been observed? 

11. Have quality control or monitoring 
procedures for data gathering and 
management been specified? 

12. Have analysis methods been chosen that 
are technically excellent and appropriate 
for providing ansv/ers to the evaluation 
questions? 

15. Have procedures been developed for 

reducing the evaluation information? 

14. Have strategies been devised for pilot 
testing the information analysis 
activities? 

15. Are staff, time, and money available for 
conducting the planned analysis? 

16. Are the results of the analysis inter- 
preted to within the scope of the 
available information? 
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Question 



Response 
Yes No 



17* Are plans made for filing and storing 
information for possible future use? 

18, Is the report designed so that parents, 
teachers, administrators, employers, and 
funding agencies can read it and use the 
information in it? 

19, Does the report contain the answers to 
the evaluation questions? 

20, Does the report explain clearly and in 
svirficient detail an explanation of the 
procedures used to arrive at the ansv/ers 
to the evaluation questions? 

21m Is the evaluation being conductefl 
according to schedule? 

22. Have the appropriate staff been assigned 
so that each evaluation task is likely to 
be completed? 

23. Has the evaluation budget been designed 
so that the evaluation questions can all 
be answered? 

24* Has sufficient time been allocated for the 
accomplishment of each evaluation task? 
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CHAPTER IX 
Fonnat/ Samples 

A. Format for Instructional Materials 

B. Sample Materials 
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A. PORI-IAT FOR INSTRUCTIONAL MATERIALS 



!• State objectives of instruction 

The local administirator and staff should be provided with a 
statement of the objectives of instruction. The objectives should 
be stated in relatively specific terms, although they need not 
always be behavioral. Behavioral objectives are frequently 
difficult to v/rite, time-consuming, and they sometimes result in 
focusing attention to somewhat trivial types of learning. However, 
they do help to clarify -^he intent of instruction. Thus, judgment 
should be used in dett;Xijaining just how specific an objective must 
be in order to clearly communicate the instructor's intent. 

Typical objectives might be: 

• to identify the considerations involved in selecting 
evaluation questions 

• to list common types of evaluation questions. 

The purpose of providing the objectives is to give the adminis- 
trator a precise overview of the contents. 

2. Organize and sequence content so that it is likely to 
facilitate achievement of the objectives^ 

Information about evaluation activities should be presented 

so that the local administrator is provided with definitions, 

explanations, and examples' of all ideas as tiiey relate to each 

objective of instruction. It is often helpful to rely upon an 
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outline that lists each objective in the order in which it should 
"be taught. 

3. Provide review questions. 

Short questions should be offered to help the local adminis- 
trator review important ideas presented in the materials. 

4. Provide practice exercises and feedback. 

> Practice exercises should be used as a vehicle for oppor- 
timities to solve the problems they a^^e likely to encoimter in 
evaluating their programs. Peedback to each exercise should be 
given. 
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SAMPLE MATERIALS 



Ob jectiTes 

Each cooperative education program is unique. In order to deter- 
mine a program' s effectiveness, a variety of questions can be asked 
about its goals, outcomes, impacts and costs. The major purpose 
of this chapter is to enable administrators of cooperative 
education programs to: ' - 

(1) Systematically reviev/ the program's goals and activities 
as a basis for framing evaluation questions 

(2) Select and suggest the types of information that will be 
convincing to those concerned with the program as 
evidence of the program's success 

(3) Pose evaluation questions that are consistent with the 
needs and concerns of the school, employers, students, 
parents, advisory committee, and fiuiding agencies. 

Introduction 

To be effective, an evaluation must be credible to its 
audiences. That is, it must be believable to and usable by 
schools, teachers, students, employers, advisory committee members, 
and fxmding agencies. To ensure credibility, an evaluation must: 

1. Reviev/ the program's goals and activities. 

Waen reviev/ing a program's goals and activities, the evaluator 
should become familiar v/ith the interests and concerns of all 
groups who have a stake in the program. 
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2. Be responsive to the types of information that will be 
convincing as evidence of the program's success. 

There are many different ways to prove that a program has 

been successful; e.g., that its goals have been achieved or that 

it was managed well. The evaluator is responsible for determining 

which of these kinds of info3?raation will provide the most believable 

evidence of the program's success to the individuals v;ho must use 

the evaluation's findings. 

3. Pose specific questions that the evaluation's audiences want 
ansv/ered. 

The evaluator must be responsive to the audience's information 
needs • To do this, evaluations ohould be designed to give answers 
to questions of concern and to provide information about program 
success. 

4. Make sure that those participating in the evaluation understand 
its procedures and products. 

The evaluator must make sure that participants understand 

v/hat an evaluation is, why it is being conducted, and hov/ 

evaluation information v/ill be presented. 

The Evaluator 's Description of the Program 

A credible evaluation incorporates into it the evaluator 's 
knov/ledge of the program and the types of information that provide 
evidence of the program's success. The Evaluator' s Description is 
a convenient form for the evaluator to use for recording and 
summarizing information about a cooperative education program's 
goals, the activities to be undertaken to achieve those goals, and 

108 



the types of evidence that must be provided to establish that each 
program goal has been achieved. An illustration of an Evaluator's 
Description Form follov/s: 

Form 1: Evaluator's Description 



Goals ActJ 



Evidence of 
Program Success 



(One by one, (For ea (For each goal, 

describe the goals describu ti'ram describe the type o^, 

the program, cr a activities that will information that will 
component of it, is lead to the fulfill- be convincing evi- 
aiming to achieve.) ment of the goal.) dence of program 

merit. ) 

Columns 1 and 2 of the Evaluator^s Description: "Goals^* and 
"Activities" 

A program goal is a statement of intent. An activity is a 

means of achieving a goal. In completing the Evaluator's 

Description, the evaluator must describe the one or more activities 

that are planned to accomplish each goal. The goals and activities 

columns of an Evaluator 's Description could take the following 

form for a component of a cooperative education program: 

Evaluator 's Description of a Cooperative 
Office Education Program (CGS) 



Goals 



A 4.- •4.-;^o Evidence of 

Activities ^ ^ 

Program Success 



1. Upon completion la<» Study standard 
of the program., office procedures 

the students v/ill in related class 

have mastered 

basic office l"b. Observe and 

procedures participate in 

procedures at wori 

station 

Some programs carefully delineate their goals and activities 
and the relationship they share. Unfortunately, not all programs 
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contain clear and complete statements of their goals and activities, 
and relationships between goals and activities are often difficult 
to determine. In the latter case, the evaluator must use experi- 
ence and logic to extract clarified goals anu activities. 

The evaluator 's job in formulating program goals and 
activities is to state them in such a nann ''at: 

• all important components of the pr^^rajn are represented 

• each goal has one or more activities associated with it 

• the priority of the goals is made clear 

• the resulting number of goals and activities is manageable 
for evaluation purposes 

Review Questions 

1. List four criteria for ensuring a credible evaluation. 

2. V/hat is meant by a program's '*goals"? Its "activities"? 

3. V/hat are the three components of the Evaluator 's Description? 



Column 3 of The Evaluator 's Description; Evidence of Program Success 

In column 3, the types of information that are acceptable as 
evidence of a program *s success should be sununarized. If the 
evaluator is only interested in one component of a cooperative 
education program, say its health occupations component, then the 
evaluator would just have to concentrate on gathering evidence of 
success for that single component. 

A program is successful if its goals are attained; if its 
activities achieve these goals and are at the same time inherently 

no 
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beneficial; and there are no vmpleasant convsequences associated 
with the program. The actual evidence that a program is successful 
will take the form of statements, events, objects, and observations 
that testify to its quaD.ity. For example, in an office occupations 
component of a cooperative education program that has as one of its 
goals to improve typing skills, credible evidence of success might 
include any or all the iwing: 

. a measured gaj, in uents' typing skills 

• an observed gain in students' typing skills 

• testimony from student-trainees that their skills have 
improved 

• testimony from parents that their children's skills have 
improved 

• testimony from teachers that their students' skills have 
improved 

• testimony from employers that student-trainees' skills have 
improved 

At least one indication of program success must be identified 
for each program goal an.d activity. For example, in a program in 
auto body repair, the evidence of program success may be illus- 
trated as follov/s: 
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Evaluator's Description of a Program 
in Auto Body Repair 



Goals 



Activities 



Evidence of 
Program Success 



!• the student v/ill 
be able to esti- 
mate damage and 
compute labor 
and material 
costs 



la. practice esti- la. 
mation of 
damages on 
dajnaged cars 
brought into 
shop 

lb. 

lb. practice 

computation of 
labor and 
material costs 
in related class 



Students can 
consistently 
estimate damages 
within accept- 
able error limits. 

Students can 
demonstrate 
ability to 
compute labor 
and material 
costs. 



To be believable, the information that would be convincing 
evidence of the program's success should be developed cooperatively 
with as many concerned persons as* possible in advance of the 
evaluation. Finding out what will convince students or employers, 
for instance, of the program's success is an extremely important 
component of a credible evaluation because it directs the evaluator 
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to identify the information that must be produced, and it forces 
the evaluation's participants to state in truthful and realistic 
tems what they rt^ally want to know about the program. Agreeing 
on evidence can be a very important safeguard for both the 
evaluator and the participants because it protects against 
assertions that the evaluation findings are not relevant or not 
sufficient to prove the program's success or failure, and it 
protects the participant against the evaluator 's arbitrarily 
collecting information jiaimed to be "good" or "important." 

Completin/y the Evaluator' s Description 

The first draft of the Evaluator' s Description is best 
prepared by the evaluator working alone, basing his or her ideas 
on knowledge of the program and a thorough review of the program's 
documentation. (No matter how well a program administrator may Icnow 
a program it is still advisable to check perceptions with the 
written word.) A program's documents, it should be noted, vary 
greatly from program tc program with respect to their number, 
quality, and format. Some new programs may only have planning 
memos or outlines and perhaps a proposal* Other, more established 
programs, may have volumes of reports. Every attempt must be made 
to use whatever documents are available to extract a list of 
program goals and activities. This list should be considered a 
first effort at developing the Evaluator 's Description, and some- 
times remain incomplete. It will, however, provide the evaluator 
with a sense of the program's direction, aims, comprehensiveness, 
and content. Further, it will facilitate subsequent discussions 
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with participants by providing everyone v;ith a uniform framework 
and vocabulary to use for understanding the evaluation. 

Upon completion, a preliminary draft of the Evaluator's 
Description should be shown to a sample of the evaluation's 
participants for review and amendments. Depending upon the 
nature, purpose, and size of the program, this document will 
xaidergo as many changes as necessary to satisfy all persons 
concerned. 

Once the evaluator has a complete description of the program's 
goals, activities, and evidence of success, and the description has 
been accepted by the evaluation's participants as a basis for the 
evaluation, then the Evaluation's Description' is complete. The 
final version of the Evaluator 's Description should be typed and 
made available to all individuals who participated in its 
development. 

Reviev/ Question 

What types of evidence might convince all potential partici- 
pants that the following possible cooperative education goals have 
been met: 

• Can follow both written and verbal directions 

• Shows a great deal of initiative 

• Develop in the student individual qualifications for 
subsequent full time employment or advanced study. 
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The Evaluation Questions 

Evaluation q.uestions can take the following general forms: 

(1) To what extent were the program's goals achieved? 

(2) Were the program's activities implemented as planned? 

(3) How effective were these activities in achieving the goals? 

(4) Por which groups was the program most/least successful? 

(5) How did internal and external social and political factors 
influence the program's development and impact? 

(6) What social and political effects did the program have on 
the environment in which it was implemented? 

(7) What did the program cost? 

(8) How well v/as the '. jogram managed? 

The evaluation qviestions that will be of concern will vary: 
in one evaluation, the questions might only "be related to the 
program's goals and activities, while in another, they might 
focus only on costs* In any case, the number of questions that 
can be answered depends upon the money, time, and resources 
available for evaluation* 

The evaluation questions are the heart of the evaluation, arxd 
all evaluation activities must be organized so that they can be 
answered efficiently* Because of the importance of the questions, 
the evaluation's client should agree to their selection and 
statement* 

Formularting the Evaluation Questioiis 

Wmp^ foundation for the evaluation questions should be the 
pro grasiL success column of the Evaluator's Description* Consider 
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the excerpted Evaluator's Description for the program in auto 
body repair given on page 4D . From the "evidence of program 
success" column, the evaluator could extract three specific 
evaluation questions, based on the larger one: can students . ~~ 

consistently estimate damage and compute costs? 

1. Are the estimates made by students within an acceptable 
error range? 

... Are the students' computations of labor costs correct? 

3, Are the students' computations of cost of material correct? 

For many evaluations, particularly those with small budgets 
conducted in an improvement context, evaluation questions can be 
drawn directly from the program success column. However, additional 
evaluation questions may also be asked. For example, for the same 
auto body repair program, the following questions could be important: 

1. V/hich student-trainees profited most from the program? 
Which ones profited least ? 

2. How effective would the program be if its funds v^ere 
reduced by 20%? 

In some evaluations, the school, students, parents, employers, 
and evaluator have no difficulties vrarking together to arrive at 
the evaluation questions and these people's cooperation is assured 
thrciaghout. In others, some or all questions may have been posed 
well in advance of the actual evaluation, say at the time of 
■pvo-SEsm plaiming. In this situation, the evaluator should check 
to ses that the questions are still thought to be important, and 
that mo new ones need to be added. Finally, the evaluation 

116 

-100- 

o 

ERIC 



questions are sometimes fixed in advance, say by the legislation 
creating cooperative education programs, and although the evaluator 
may choose to make additions, he or she is bound to at least 
answer the mandated ones. 

Regardless of the o-ri p;j i oi Ai^^ questions, ^.M/^ui-u... , 

should get approval of the completed list of evaluation questions. 
Thus, once a preliminary draft is available it should be sent for 
a review that should serve to: 

• add or delete questions 

• establish priorities for questions 

• limit or enlarge the number of questions 

• check that all questions can be addressed within the time 
period, given the resources available. 

Review Questions 

1. List 8 general evaluation questions. _ ,.: 

2. Explain why the proposed final draft of evaluation questions 
should be submitted for review. 

A Note on Field Testin/a; the Evaluation Handbook 

To be effective, preliminary versions of all instructional 
materials should be tried out with a sample of administrators of 
cooperative education and their staffs to help determine whether 
the instructional objectives are clearly and appropriately stated, 
the sequence and organization of content are correct, and the 
practice exercises and feedback are worthwhile. 
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r^XERCISE 



Directions; 

In this exercise, you are given a portion of a conversation 
among an evaluator, a teacher-coordinator, and an employer concern 
ing a cooperative education program in architectural drafting. 
These three individuals are discussing possible evaluation 
questions. Your task is to compile a v/ritten list of evaluation 
questions based on the conversation. 

Write your answers on the remainder of this page. The 
conversation can be foimd on the next page. 

Evaluation Questions 
(Write answers below) 
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Conversation 



Evaluator 

The purpose of this meeting is to discuss evaluation questions 
for the architectural drafting program. 

Teacher-Coordinator 

I have been thinking about some questions. They are: 

1. Can students describe the company pay policies at their work 
sites? 

2. Can students explain the organization of the firm in which 
they are employed? 

Employer 

I don't think the second question is a good one. Some 
companies in the program are very large, and their structure and 
organization are complicated. This is true of our firm. If the 
student remains with us for an extended period of time, the question 
might become important. Therefore, I think the first question is 
a much more important one for an entry-level employee. Let me add 
three questions v/hich deal with the student-trainee's skills: 

1. Can students make accurate ink drawings? 

2. How efficiently and accurately do students use drafting tools? 

3. How neatly and rapidly do students letter? 

I would also like a question to be included that concerns 
students' attitudes toward improving the neatness of their work. 

119 



-103- 



Teacher-Coordinator 

At our last meeting you asked a question about v/hether 
employees are familiar with the insurance needed to protect the 
student-trainee, v/hich I think is -very important and v/ould like 
to keep. I would also like to suggest a question concerning 
whether or not the classroom materials provided by the coopera- 
tive education office contain clear statements of purposes that 
student-trainees can easily understand. Some of the materials 
I've reviewed rely on very long lists of objectives that use 
complicated words. 

Evaluator 

I v/ill attempt to revise the questions according to your 
suggestions, but I am afraid that our budget might not allow us 
to go through all the instructional materials at the cooperative 
education office, and I might therefore have to take the question 
off the list. In the meantime, I will keep it and notify you if 
any changes are necessary. 
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Answers to Exercise 
Evaluation Questions 

!• Can students describe company pay policies? 
2. Can students make accurate ink drawings? 

5. How efficiently and accurately do students use drafting tools? 
4- How neatly and rapidly do students letter? 

5. To what extent have students' attitudes changed with respect 
to improving the neatness of their work? 

6. Are employers familiar with the insurance needed to protect 
students? 

?• Do classroom materials contain statements of purposes that 
students can easily vinderstand? 
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